Self Regulation Ontology Retest Analyses
Introduction
The psychological construct of self-regulation, or related concepts of impulsivity, self-control, inhibition and others correlate with problematic real-world behaviors such as disordered eating (Mobbs et al., 2010, Verbeken et al., 2009, Nederkoorn et al. 2006a, Nederkoorn et al., 2006b), gambling (Lawrence et al., 2009, Fuentes et al., 2006, Alessi and Petry, 2003), drug addiction (Coffey et al., 2003, de Wit, Crean and John, 2000, Sher, Bartholow and Wood, 2000, Kirby, Petry and Bickel, 1999) or bad financial decisions (Meier and Sprenger, 2015, Meier and Sprenger, 2013, Meier and Sprenger 2012). In these lines of work measures of self-regulation are used as behavioral assays for individual difference analyses. An underlying assumption of treating behavioral measures as reflective of person-specific characteristics is that the measures of self-regulation are stable across time. In other words, that they have high test-retest reliability (or high between- and low within-subject variability). This assumption is not extensively and equally tested in the literature for all measures of self-regulation.
Self-regulation measures can be grouped in two broad categories: Surveys and cognitive tasks. While the former typically goes through some form of psychometric testing often filtering out items with low test-retest reliability this is less frequently true for the latter. Though some empirical results on test-retest reliabilities of certain cognitive task measures are reported in pockets of the literature an exhaustive summary or systematic elimination of tasks or measures based on stability across time does not seem to exist. Test-retest reliability is, however, crucial for individual difference analyses (Hedge, Powell and Sumner, 2017).
To answer the question of reliability of self-regulation measures comprehensively we created a large battery consisting of both cognitive task and survey measures. In this paper we make two contributions: First we provide a review of these measures along with their reported reliabilities when available. Then we report results of a large study we conducted where we collected retest reliability data using all of these measures. This effort not only fills in a notable gap in the literature in clarifying which measures of self-regulation are stable across time but also provides guidance on factors to pay attention to when constructing new tasks for individual difference measures.
Methods
Data collection
This sample consists of data for 150 subjects of the original sample of 522 that has completed the initial battery of 37 cognitive tasks, 23 surveys and 3 surveys on demographics. Details of the original sample as well as quality control (qc) procedures are described elsewhere (Eisenberg et al., 2017). Invited participants were chosen randomly and only subsets of them were invited for a given batch (instead of opening the battery to all qualified subjects) with the intention to avoid a potential oversampling and bias towards “high self regulators”.
workers = read.csv(paste0(retest_data_path,'Local/User_717570_workers.csv'))
workers = workers %>%
group_by(Worker.ID) %>%
mutate(Retest_worker=ifelse(sum(CURRENT.RetestWorker,CURRENT.RetestWorkerB2,CURRENT.RetestWorkerB3,CURRENT.RetestWorkerB4,CURRENT.RetestWorkerB5,na.rm=T)>0,1,0)) %>%
ungroup()
worker_counts <- fromJSON(paste0(retest_data_path,'/Local/retest_worker_counts.json'))
worker_counts = as.data.frame(unlist(worker_counts))
names(worker_counts) = "task_count"In total 242 participants were invited, 175 began the battery, 157 completed the battery and 150 provided data that passed qc for both time points. Our target sample size was determined in advance of data collection and data collection continued until this number of participants with data that survived qc was reached.
disc_comp_date = read.csv(paste0(retest_data_path,'Local/discovery_completion_dates.csv'), header=FALSE)
val_comp_date = read.csv(paste0(retest_data_path,'Local/validation_completion_dates.csv'), header=FALSE)
test_comp_date = rbind(disc_comp_date, val_comp_date)
rm(disc_comp_date, val_comp_date)
retest_comp_date = read.csv(paste0(retest_data_path,'Local/retest_completion_dates.csv'), header=FALSE)
comp_dates = merge(retest_comp_date, test_comp_date, by="V1")
names(comp_dates) <- c("sub_id", "retest_comp", "test_comp")
comp_dates$retest_comp = as.Date(comp_dates$retest_comp)
comp_dates$test_comp = as.Date(comp_dates$test_comp)
comp_dates$days_btw = with(comp_dates, retest_comp-test_comp)Data collection took place on average 115 number of days after the completion of the initial battery with a range of 60 to 228 days.
rm(test_comp_date, retest_comp_date, comp_dates)Demographics
test_demog <- read.csv(paste0(retest_data_path, '/t1_data/demographic_health.csv'))
retest_demog <- read.csv(paste0(retest_data_path, 'demographic_health.csv'))
retest_demog = retest_demog[retest_demog$X %in% test_demog$X,]
names(test_demog)[which(names(test_demog) == 'X')] <-'sub_id'
names(retest_demog)[which(names(retest_demog) == 'X')] <-'sub_id'
summary(test_demog %>%
select(Sex, Age)) Sex Age
Min. :0.00 Min. :21.0
1st Qu.:0.00 1st Qu.:28.0
Median :1.00 Median :33.0
Mean :0.52 Mean :34.1
3rd Qu.:1.00 3rd Qu.:38.8
Max. :1.00 Max. :59.0 summary(retest_demog %>%
select(Sex, Age)) Sex Age
Min. :0.000 Min. :21.0
1st Qu.:0.000 1st Qu.:29.0
Median :1.000 Median :33.0
Mean :0.527 Mean :34.5
3rd Qu.:1.000 3rd Qu.:38.8
Max. :1.000 Max. :60.0 Literature
One of the major contributions of this project is a comprehensive literature review of the retest reliabilities of the surveys and tasks that were used. We reviewed the literature on a measure (as opposed to task level) paying attention to differences in sample size, the delay between the two measurements as well as the statistic that was used to assess reliabilities (e.g. Spearman vs. Pearson correlations). Here we present a table and a visualization summarizing our findings. References mentioned in the table below can be found here.
lit_review <- read.csv('/Users/zeynepenkavi/Dropbox/PoldrackLab/SRO_Retest_Analyses/input/lit_review_figure.csv')
lit_reviewlit_review = lit_review %>%
separate(dv, c("task_group", "var"), sep="\\.",remove=FALSE,extra="merge") %>%
mutate(task_group = factor(task_group, levels = task_group[order(task)]),
type = as.character(type)) %>%
mutate(task_group = gsub("_", " ", task_group),
var = gsub("_", " ", var)) %>%
arrange(task_group, raw_fit, var) %>%
mutate(task_group = ifelse(task_group == "psychological refractory period two choices", "psychological refractory period", ifelse(task_group == "angling risk task always sunny", "angling risk task",task_group))) %>%
mutate(task_group = gsub("survey", "", task_group)) %>%
select(-measure_description)Warning in `levels<-`(`*tmp*`, value = if (nl == nL) as.character(labels)
else paste0(labels, : duplicated levels in factors are deprecatedtmp = lit_review[duplicated(lit_review$reference)==FALSE,]
nrow(tmp)[1] 154sum(tmp$sample_size)[1] 17550nrow(lit_review)[1] 583rm(tmp)Measure level plot
lit_review = lit_review %>%
select(-reference)
Because this plot is difficult to digest we summarize it on a task level to give a general sense of the main takeaways. This plot naturally disregards much of the fine grained information.
p1_t_legend <- lit_review %>%
filter(task == 'task') %>%
ggplot(aes(y = factor(task_group, levels=rev(unique(task_group))), x = retest_reliability)) +
geom_point(aes(size=sample_size, shape = type), color='black')+
theme(axis.text.y = element_text(size=43),
legend.position = 'bottom',
axis.text.x = element_text(size=23),
axis.title.x = element_text(size=30),
legend.text = element_text(size=20),
legend.key.width = unit(0.75, "inches"),
legend.title = element_text(size=28),
legend.spacing.x = unit(0.5, "inches")) +
guides(size = guide_legend(override.aes = list(size=c(9,18,28))),
shape = guide_legend(override.aes = list(size=18)))+
xlab("Retest Reliability")+
ylab("")+
scale_x_continuous(limits = c(-0.25,1), breaks=c(-0.25, 0, 0.25, 0.5, 0.75, 1))+
scale_y_discrete(labels = function(x) str_wrap(x, width = 10))+
scale_shape_manual(breaks = sort(lit_review$type), values = c(15, 16, 17, 3), name="Type")+
scale_size_continuous(name = "Sample Size")+
geom_vline(xintercept = 0, color = "red", size = 1)
p1_t <- lit_review %>%
filter(task == 'task') %>%
ggplot(aes(y = factor(task_group, levels=rev(unique(task_group))), x = retest_reliability)) +
geom_point(aes(size=sample_size, shape = type), color='#00BFC4')+
theme(axis.text.y = element_text(size=43),
legend.position = 'none',
axis.text.x = element_text(size=23),
axis.title.x = element_text(size=30)) +
xlab("Retest Reliability")+
ylab("")+
scale_x_continuous(limits = c(-0.25,1), breaks=c(-0.25, 0, 0.25, 0.5, 0.75, 1))+
scale_y_discrete(labels = function(x) str_wrap(x, width = 10))+
scale_shape_manual(breaks = sort(lit_review$type), values = c(15, 16, 17, 3))+
scale_size_continuous(range = c(5, 35))+
geom_vline(xintercept = 0, color = "red", size = 1)
p2_t <- lit_review %>%
filter(task == 'survey') %>%
ggplot(aes(y = factor(task_group, levels=rev(unique(task_group))), x = retest_reliability)) +
geom_point(aes(size=sample_size, shape = type), color='#F8766D')+
theme(axis.text.y = element_text(size=43),
legend.position = 'none',
axis.text.x = element_text(size=23),
axis.title.x = element_text(size=30)) +
xlab("Retest Reliability")+
ylab("")+
scale_x_continuous(limits = c(-0.25,1), breaks=c(-0.25, 0, 0.25, 0.5, 0.75, 1))+
scale_y_discrete(labels = function(x) str_wrap(x, width = 10))+
scale_shape_manual(breaks = sort(lit_review$type), values = c(15, 16, 17, 3))+
scale_size_continuous(range = c(5, 35))+
geom_vline(xintercept = 0, color = "red", size = 1)
mylegend<-g_legend(p1_t_legend)
p3_t <- arrangeGrob(arrangeGrob(p1_t, p2_t, nrow=1), mylegend, nrow=2,heights=c(10, 1))
ggsave('Lit_Review_Plot_t.jpg', plot = p3_t, device = "jpeg", path = "/Users/zeynepenkavi/Dropbox/PoldrackLab/SRO_Retest_Analyses/output/figures/", width = 27, height = 48, units = "in", limitsize = FALSE, dpi = 72)
rm(p1_t, p2_t, p3_t, mylegend, p1_t_legend)
An interactive version of this plot could be find here
Takeaways from this review are:
- Survey measures have been reported to higher reliability compared to task measures
- Survey measures have less variability in the reported reliabiltiy estimates compared to task measures
Loading datasets
The variables included in this report are:
- meaningful variables (includes only hdddm parameters)
- EZ diffusion parameters
- Raw RT and Accuracy measures
- Variables found in the literature (for comparison)
Load time 1 data
test_data <- read.csv(paste0(retest_data_path,'t1_data/variables_exhaustive.csv'))
test_data <- test_data[,names(test_data) %in% retest_report_vars]
test_data$X <- as.character(test_data$X)
names(test_data)[which(names(test_data) == 'X')] <-'sub_id' For reference here are the variables that are not included in the analyses of the remainder of this report because they were not of theoretical interest in factor structure analyses of this data so far. These include drift diffusion and other model parameters for specific conditions within a task; survey variables that are not part of the dependant variables for that survey in the literature and demographics (these are saved for prediction analyses).
test_data2 <- read.csv(paste0(retest_data_path, 't1_data/variables_exhaustive.csv'))
df <- data.frame(names(test_data2)[which(names(test_data2) %in% names(test_data) == FALSE)])
names(df) = c('vars')
dfLoad time 2 data
retest_data <- read.csv(paste0(retest_data_path,'variables_exhaustive.csv'))
retest_data <- retest_data[,names(retest_data) %in% retest_report_vars]
retest_data$X <- as.character(retest_data$X)
names(retest_data)[which(names(retest_data) == 'X')] <-'sub_id'
retest_data = retest_data[retest_data$sub_id %in% test_data$sub_id,]Replace HDDM parameters in t1 data
Since HDDM parameters depend on the sample on which they are fit we refit the model on t1 data for the subjects that have t2 data. Here we replace the HDDM parameters in the current t1 dataset with these refitted values.
hddm_refits <- read.csv(paste0(retest_data_path,'t1_data/hddm_refits_exhaustive.csv'))
hddm_refits = hddm_refits[,names(hddm_refits) %in% retest_report_vars]
hddm_refits$X <- as.character(hddm_refits$X)
names(hddm_refits)[which(names(hddm_refits) == 'X')] <-'sub_id'
#For later comparison of whether fitting the DDM parameters on full or retest sample makes a big difference
test_data_full_sample_hddm <- test_data
#drop hddm columns from test_data
test_data = cbind(test_data$sub_id, test_data[,names(test_data) %in% names(hddm_refits) == FALSE])
#fix naming before merging
names(test_data)[which(names(test_data) == 'test_data$sub_id')] <-'sub_id'
#merge hddm refits to test data
test_data = merge(test_data, hddm_refits, by="sub_id")Results
Data quality checks
Demographics reliability
Point estimates of reliability for the demographic variabels.
numeric_cols = c()
for(i in 1:length(names(test_demog))){
if(is.numeric(test_demog[,i])){
numeric_cols <- c(numeric_cols, names(test_demog)[i])
}
}
demog_rel_df <- data.frame(spearman = rep(NA, length(numeric_cols)),
icc = rep(NA, length(numeric_cols)),
pearson = rep(NA, length(numeric_cols)))
row.names(demog_rel_df) <- numeric_cols
for(i in 1:length(numeric_cols)){
demog_rel_df[numeric_cols[i], 'spearman'] <- get_spearman(numeric_cols[i], t1_df = test_demog, t2_df = retest_demog)
demog_rel_df[numeric_cols[i], 'icc'] <- get_icc(numeric_cols[i], t1_df = test_demog, t2_df = retest_demog)
demog_rel_df[numeric_cols[i], 'pearson'] <- get_pearson(numeric_cols[i], t1_df = test_demog, t2_df = retest_demog)
}
demog_rel_df = demog_rel_df %>%
mutate(var = row.names(.)) %>%
select(var, icc, spearman, pearson) %>%
arrange(-icc)
demog_rel_dfsjt.df(demog_rel_df %>% mutate_if(is.numeric, funs(round(., 3))), describe=F, hide.progress = TRUE, show.rownames = FALSE, file = "/Users/zeynepenkavi/Dropbox/PoldrackLab/SRO_Retest_Analyses/output/tables/demog_rel_table.doc")| var | icc | spearman | pearson |
|---|---|---|---|
| HispanicLatino | 1 | 1 | 1 |
| Age | 0.998 | 0.996 | 0.997 |
| Sex | 0.993 | 0.987 | 0.987 |
| HouseholdIncome | 0.988 | 0.955 | 0.977 |
| ArrestedChargedLifeCount | 0.958 | 0.829 | 0.92 |
| HighestEducation | 0.951 | 0.895 | 0.908 |
| WeightPounds | 0.951 | 0.936 | 0.907 |
| ChildrenNumber | 0.95 | 0.971 | 0.907 |
| HowLongSmoked | 0.949 | 0.877 | 0.904 |
| RelationshipNumber | 0.948 | 0.9 | 0.901 |
| CigsPerDay | 0.945 | 0.841 | 0.895 |
| LifetimeSmoke100Cigs | 0.944 | 0.894 | 0.894 |
| AlcoholHowManyDrinksDay | 0.936 | 0.871 | 0.881 |
| MortgageDebt | 0.93 | 0.833 | 0.87 |
| RetirementPercentStocks | 0.928 | 0.863 | 0.867 |
| LongestRelationship | 0.928 | 0.853 | 0.867 |
| HowOftenFailedActivitiesDrinking | 0.927 | 0.782 | 0.866 |
| CoffeeCupsPerDay | 0.926 | 0.876 | 0.863 |
| RentOwn | 0.926 | 0.862 | 0.862 |
| HowOftenMemoryConcentrationProblemCannabis | 0.925 | 0.594 | 0.863 |
| SmokeEveryDay | 0.916 | 0.802 | 0.846 |
| HeightInches | 0.907 | 0.888 | 0.83 |
| DivorceCount | 0.905 | 0.935 | 0.836 |
| CannabisHowOften | 0.903 | 0.856 | 0.823 |
| EducationDebt | 0.901 | 0.837 | 0.821 |
| HowOftenDevotedTimeCannabis | 0.898 | 0.814 | 0.814 |
| MedicalProblemsDueToDrugUse | 0.888 | 0.814 | 0.814 |
| AlcoholHowOften | 0.881 | 0.788 | 0.79 |
| RelationshipStatus | 0.87 | 0.788 | 0.77 |
| HowSoonSmokeAfterWaking | 0.868 | 0.787 | 0.767 |
| RetirementAccount | 0.86 | 0.756 | 0.756 |
| AlcoholHowOften6Drinks | 0.853 | 0.766 | 0.747 |
| CreditCardDebt | 0.846 | 0.808 | 0.733 |
| RelativeFriendConcernedDrinking | 0.844 | 0.71 | 0.733 |
| CannabisPast6Months | 0.84 | 0.725 | 0.725 |
| TeaCupsPerDay | 0.834 | 0.633 | 0.715 |
| HowOftenCantStopDrinking | 0.833 | 0.704 | 0.716 |
| CaffienatedSodaCansPerDay | 0.832 | 0.711 | 0.723 |
| HowOftenGuiltRemorseDrinking | 0.829 | 0.564 | 0.708 |
| CannabisHoursStoned | 0.825 | 0.698 | 0.703 |
| DoctorVisitsLastMonth | 0.815 | 0.351 | 0.701 |
| HowOftenDrinkMorning | 0.809 | 0.39 | 0.692 |
| CannabisConsideredReduction | 0.806 | 0.638 | 0.675 |
| NeurologicalDiagnoses | 0.795 | 0.66 | 0.66 |
| HowOftenUnableRememberDrinking | 0.786 | 0.61 | 0.651 |
| FeelBadGuiltyDrugUse | 0.782 | 0.642 | 0.642 |
| TrafficAccidentsLifeCount | 0.781 | 0.585 | 0.643 |
| HowOftenCantStopCannabis | 0.748 | 0.732 | 0.598 |
| InjuredDrinking | 0.734 | 0.626 | 0.592 |
| CarDebt | 0.716 | 0.698 | 0.559 |
| DaysHalfLastMonth | 0.715 | 0.552 | 0.559 |
| Worthless | 0.709 | 0.632 | 0.549 |
| DaysPhysicalHealthFeelings | 0.705 | 0.533 | 0.545 |
| Depressed | 0.689 | 0.561 | 0.527 |
| EverythingIsEffort | 0.682 | 0.553 | 0.517 |
| Hopeless | 0.678 | 0.579 | 0.516 |
| OtherDrugs | 0.658 | 0.492 | 0.492 |
| RestlessFidgety | 0.621 | 0.507 | 0.451 |
| TrafficTicketsLastYearCount | 0.621 | 0.438 | 0.45 |
| Nervous | 0.615 | 0.473 | 0.445 |
| NeglectedFamilyDrugUse | 0.488 | 0.342 | 0.342 |
| DaysLostLastMonth | 0.475 | 0.604 | 0.312 |
| EngagedInIllegalActsToObtainDrugs | 0.468 | 0.306 | 0.306 |
| AbleToStopDrugs | 0.458 | 0.306 | 0.306 |
| BlackoutFlashbackDrugUse | 0.424 | 0.272 | 0.272 |
| HowOftenFailedActivitiesCannabis | 0.423 | 0.636 | 0.378 |
| WidthdrawalSymptoms | 0.403 | 0.254 | 0.254 |
| Last30DaysUsual | 0.395 | 0.285 | 0.248 |
| GamblingProblem | 0.317 | 0.38 | 0.195 |
| AbuseMoreThanOneDrugAtATime | 0.294 | 0.173 | 0.173 |
| HowOftenHazardousCannabis | 0.057 | 0.369 | 0.029 |
| CaffieneOtherSourcesDayMG | 0 | 0.363 | -0.033 |
| SpouseParentsComplainDrugUse | -0.033 | -0.017 | -0.017 |
| OtherDebtAmount | -0.083 | 0.005 | -0.041 |
summary(demog_rel_df) var icc spearman pearson
Length:74 Min. :-0.0831 Min. :-0.0166 Min. :-0.0406
Class :character 1st Qu.: 0.6793 1st Qu.: 0.5373 1st Qu.: 0.5161
Mode :character Median : 0.8323 Median : 0.7006 Median : 0.7156
Mean : 0.7417 Mean : 0.6558 Mean : 0.6446
3rd Qu.: 0.9257 3rd Qu.: 0.8401 3rd Qu.: 0.8628
Max. : 1.0000 Max. : 1.0000 Max. : 1.0000 Effect of delays between the two measurements
Due to our data collection strategy we did not strictly control for the delay between the two measurements as standard psychometrics studies measuring retest reliability might.
An individual difference measure would preferably remain stable regardless of the delay between multiple measurements.
Since we only had two measurements we could not test directly whether a measure becomes less reliable depending on the delay between the two time points (The average number of days between two measurements would be the same for all measures since reliability is a measure level metric while days between completion a subject level one).
So if you regress the average difference for each measure on average delay between two measurements you are regressing a vector with varying numbers on a vector of same values, like a t test asking if the mean of the varying column, in this case the average difference score, is different than the unique value in the single value column, i.e. the average delay between two time points (Note that this should be true in theory but in practice since for each measure there might subjects for whom the dv could not be calculated there might be >1 unique values for the average delay between the two time points). This analysis is not meaningful.
Instead of the summary metric like the retest reliability estimate for each measure we can check whether the difference score distribution for each measure depends on the delay between the two measurements. Since the difference score distribution is at subject level we can check whether the order of subjects in this distribution depends on their order in the distribution of days between completing the two tests.
Make data frame with difference between two scores for each measure for each subject. Since the scores for different measures are on different scales for comparability the difference scores are normalized (ie demeaned and dividied by the sd of the difference score distribution.) Note that the variance of the difference score distribution accounts for the variance in both time points by summing them. Normalization equates the means of each difference score distribution to 0 which would mask any meaningful change between the two time points but the analysis here does not interpret the mean of the difference score distributions but is interested in its relation to the days between completion. We check if the variables show systematic differences between the two points later.
Here we check if the difference is larger the longer the delay
numeric_cols = c()
for(i in 1:length(names(test_data))){
if(is.numeric(test_data[,i]) & names(test_data)[i] %in% names(retest_data)){
numeric_cols <- c(numeric_cols, names(test_data)[i])
}
}
t1_2_difference = data.frame()
for(i in 1:length(numeric_cols)){
tmp = match_t1_t2(numeric_cols[i],format='wide')
tmp = tmp %>%
mutate(difference = scale(`2` - `1`))
t1_2_difference = rbind(t1_2_difference, tmp)
}
t1_2_difference$difference = as.data.frame(t1_2_difference$difference)$V1
t1_2_difference = t1_2_difference %>% separate(dv, c("task", "dv2"), sep="\\.", remove=FALSE)Add completion dates to this data frame.
retest_task_comp_times = read.csv(paste0(retest_data_path, 'Local/retest_task_completion_times.csv'))
test_task_comp_times = read.csv(paste0(retest_data_path, 'Local/test_task_completion_times.csv'))
task_comp_times = merge(retest_task_comp_times, test_task_comp_times, by=c('worker_id','task'))
rm(retest_task_comp_times, test_task_comp_times)
task_comp_times = task_comp_times %>%
select(-X.x, -X.y) %>%
mutate(finish_day.x = as.Date(finish_day.x),
finish_day.y = as.Date(finish_day.y),
days_btw = finish_day.x-finish_day.y) %>%
rename(sub_id=worker_id)t1_2_difference = merge(t1_2_difference, task_comp_times[,c('sub_id', 'task','days_btw')], by=c('sub_id', 'task'))What does the distribution of differences look like: The distribution of differences between two time points for each measure
t1_2_difference %>%
ggplot(aes(difference, alpha=dv))+
geom_histogram(position='identity')+
theme(legend.position = 'none')
How do the difference score distributions look like with respect to the days between completion?
t1_2_difference %>%
ggplot()+
geom_smooth(aes(as.numeric(days_btw), abs(difference), group=factor(dv)), method='lm', se=FALSE)+
geom_smooth(aes(as.numeric(days_btw), abs(difference)), method='lm', color = "black", se=FALSE)+
theme(legend.title = element_blank())+
xlab('Days between completion')+
ylab('Absolute Scaled difference score')
To test if the slope of the black is significant we would run a mixed effects model with a fixed effect for days between completion, random slope for each dv depending on the days between and random intercept for each dv.
Before I was using subjects as a random effect but days between the two time points for each measure depends on subj id. What varies randomly is which dv we are looking for its distribution of differences in relation to the days between the time points. So I changed the model to have fixed effect for the days between, a random slope for (dependent variables can be differentially sensitive to th effect of days between) and a random intercept for dependent variable.
Significant fixed effect suggests that on average the longer the delay the smaller the difference.
summary(lmerTest::lmer(abs(difference) ~ scale(days_btw)+(scale(days_btw) | dv), data=t1_2_difference)) Linear mixed model fit by REML t-tests use Satterthwaite approximations
to degrees of freedom [lmerMod]
Formula: abs(difference) ~ scale(days_btw) + (scale(days_btw) | dv)
Data: t1_2_difference
REML criterion at convergence: 132273
Scaled residuals:
Min 1Q Median 3Q Max
-1.127 -0.719 -0.260 0.402 15.807
Random effects:
Groups Name Variance Std.Dev. Corr
dv (Intercept) 0.003190 0.0565
scale(days_btw) 0.000149 0.0122 1.00
Residual 0.468366 0.6844
Number of obs: 63454, groups: dv, 446
Fixed effects:
Estimate Std. Error df t value Pr(>|t|)
(Intercept) 0.72218 0.00382 457.00000 189.05 <2e-16 ***
scale(days_btw) -0.00985 0.00278 3236.00000 -3.55 0.0004 ***
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
Correlation of Fixed Effects:
(Intr)
scl(dys_bt) 0.146 But if I just run one mixed effects model then we don’t get a sense of the simple effects (how many of the variables this effect of the days between is significant and in which direction). I can run it separately for each dv to see if all difference score distributions are affected the same way depending on the days between completion.
get_delay_effect = function(df){
mod = lm(abs(difference) ~ scale(days_btw), data = df)
out = data.frame(estimate=coef(summary(mod))["scale(days_btw)","Estimate"], pval=coef(summary(mod))["scale(days_btw)","Pr(>|t|)"])
return(out)
}
sig_days_effect = t1_2_difference %>%
group_by(dv) %>%
do(get_delay_effect(.)) %>%
filter(pval<0.05)
sig_days_effectFor visualization I used to summarize the difference scores per person by looking at the average difference per task per subject and plot that against the number of days between completion. This yields multiple points for each value on x representing the average difference per task for each subject. But variables within a task could go in different directions (e.g. if patient proportion increases discount rate decreases) so this doesn’t seem like a good idea)
Instead I now plot all the data grouping by dependant variable and coloring depending on whether the difference score distribution for that variable has a significant slope when regressed over days between. The black is the main effect for the large multilevel model (this is the same plot as above colored by whether the difference score distribution for each dv has a significant slope depending on days between).
t1_2_difference %>%
mutate(sig_days_effect = ifelse(dv %in% sig_days_effect$dv, 1, 0))%>%
arrange(-sig_days_effect) %>%
ggplot()+
stat_smooth (aes(as.numeric(days_btw), abs(difference),
group=factor(dv, levels=unique(dv[order(sig_days_effect)]), ordered=TRUE),
color=factor(sig_days_effect, levels = c(1,0))),geom="line", alpha=0.5, method='lm')+
geom_smooth(aes(as.numeric(days_btw), abs(difference)), method='lm', color = "black", se=FALSE)+
theme(legend.title = element_blank())+
xlab('Days between completion')+
ylab('Scaled difference score')+
scale_color_discrete(breaks=c(0,1),
labels=c("NS", "Sig"))
ggsave('DaysBtwEffect.jpg', device = "jpeg", path = "/Users/zeynepenkavi/Dropbox/PoldrackLab/SRO_Retest_Analyses/output/figures/", width = 6, height = 4, units = "in", limitsize = FALSE, dpi = 100)Conclusion: The average difference between the scores of a measure if anything decreases with the increased delay. This fixed effect masks the simple effect for each dv. When looked at individually only 10 variables show a dependence on days between completion. While 7 of them show decreasing differences depending on days between 3 do show increasing differences. This suggests that the lack of stricter control over the days between completion of the measurement does not have a large effect on the stability of most measures.
effect of whether days between leads to a sig difference on reliability of that variable? plot coef of sig_day_effect over rel_df point estimate
Do the variables that show significant dependence on the delay between the completion times have systematically lower/higher reliability? Looking at the reliability point estimate over the size of the delay effect. There are very few variables to compare anyway but regardless doesn’t look conclusive/concerning.
#Create df of point estimate reliabilities
rel_df <- data.frame(spearman = rep(NA, length(numeric_cols)),
icc = rep(NA, length(numeric_cols)),
pearson = rep(NA, length(numeric_cols)),
partial_eta_sq = rep(NA, length(numeric_cols)),
sem = rep(NA, length(numeric_cols)),
var_subs = rep(NA, length(numeric_cols)),
var_ind = rep(NA, length(numeric_cols)),
var_resid = rep(NA, length(numeric_cols)))
row.names(rel_df) <- numeric_cols
for(i in 1:length(numeric_cols)){
rel_df[numeric_cols[i], 'spearman'] <- get_spearman(numeric_cols[i])
rel_df[numeric_cols[i], 'icc'] <- get_icc(numeric_cols[i])
rel_df[numeric_cols[i], 'pearson'] <- get_pearson(numeric_cols[i])
rel_df[numeric_cols[i], 'partial_eta_sq'] <- get_partial_eta(numeric_cols[i])
rel_df[numeric_cols[i], 'sem'] <- get_sem(numeric_cols[i])
rel_df[numeric_cols[i], 'var_subs'] <- get_var_breakdown(numeric_cols[i])$subs
rel_df[numeric_cols[i], 'var_ind'] <- get_var_breakdown(numeric_cols[i])$ind
rel_df[numeric_cols[i], 'var_resid'] <- get_var_breakdown(numeric_cols[i])$resid
}
rel_df$dv = row.names(rel_df)
row.names(rel_df) = seq(1:nrow(rel_df))
rel_df$task = 'task'
rel_df[grep('survey', rel_df$dv), 'task'] = 'survey'
rel_df[grep('holt', rel_df$dv), 'task'] = "task"
rel_df = rel_df %>%
select(dv, task, spearman, icc, pearson, partial_eta_sq, sem, var_subs, var_ind, var_resid)
# row.names(rel_df) = NULLsig_days_effect %>%
left_join(rel_df, by='dv') %>%
ggplot(aes(estimate, icc))+
geom_point()
Overlapping survery questions
Some surveys have overlapping questions. Do these correlate within and across sessions?
First determine the overlapping questions.
tmp = read.csv(gzfile(paste0(retest_data_path, 'items.csv.gz')))
tmp = tmp %>%
filter(worker == 's005') %>%
select(item_ID, item_text) %>%
mutate(item_text = trimws(as.character(item_text))) %>%
unite(item, c("item_ID", "item_text"), sep = "___")
comb = as.data.frame(t(combn(unique(tmp$item),2)))
duplicate_items = comb %>%
filter(grepl('dospert', V1)==FALSE) %>%
filter(grepl('selection_optimization', V1)==FALSE) %>%
filter(grepl('sensation_seeking', V1)==FALSE) %>%
separate(V1, c("item1_ID", "item1_text"), sep="___") %>%
separate(V2, c("item2_ID", "item2_text"), sep="___") %>%
mutate(similarity = levenshteinSim(item1_text, item2_text)) %>%
filter(similarity>0.8) %>%
select(item1_ID, item2_ID, item1_text, item2_text)
duplicate_items#surveys to read in
extract_items = c('worker',unique(with(duplicate_items, c(item1_ID, item2_ID))))
#correlations to compute:
#item1_t1 - item2_t1,
#item1_t2 - item2_t2,
#item1_t1 - item2_t2,
#item1_t2 - item2_t1
duplicate_items_data_t1 = read.csv(paste0(test_data_path, 'subject_x_items.csv'))
duplicate_items_data_t2 = read.csv(paste0(retest_data_path, 'subject_x_items.csv'))
duplicate_items_data_t1 = duplicate_items_data_t1 %>%
filter(worker %in% duplicate_items_data_t2$worker) %>%
select(extract_items)
duplicate_items_data_t2=duplicate_items_data_t2 %>%
filter(worker %in% duplicate_items_data_t1$worker) %>%
select(extract_items)
duplicate_items = duplicate_items %>%
mutate(t1_t1_cor = NA,
t2_t2_cor = NA,
t1_t2_cor = NA,
t2_t1_cor = NA,
t1_t1_polycor = NA,
t2_t2_polycor = NA,
t1_t2_polycor = NA,
t2_t1_polycor = NA)
for(i in 1:nrow(duplicate_items)){
duplicate_items$t1_t1_cor[i] = abs(cor(duplicate_items_data_t1[,c(duplicate_items$item1_ID[i])],
duplicate_items_data_t1[,c(duplicate_items$item2_ID[i])]))
duplicate_items$t2_t2_cor[i] = abs(cor(duplicate_items_data_t2[,c(duplicate_items$item1_ID[i])],
duplicate_items_data_t2[,c(duplicate_items$item2_ID[i])]))
duplicate_items$t1_t2_cor[i] = abs(cor(duplicate_items_data_t1[,c(duplicate_items$item1_ID[i])],
duplicate_items_data_t2[,c(duplicate_items$item2_ID[i])]))
duplicate_items$t2_t1_cor[i] = abs(cor(duplicate_items_data_t2[,c(duplicate_items$item1_ID[i])],
duplicate_items_data_t1[,c(duplicate_items$item2_ID[i])]))
duplicate_items$t1_t1_polycor[i] = abs(polychoric(data.frame(duplicate_items_data_t1[,c(duplicate_items$item1_ID[i])],
duplicate_items_data_t1[,c(duplicate_items$item2_ID[i])]))$rho[2])
duplicate_items$t2_t2_polycor[i] = abs(polychoric(data.frame(duplicate_items_data_t2[,c(duplicate_items$item1_ID[i])],
duplicate_items_data_t2[,c(duplicate_items$item2_ID[i])]))$rho[2])
duplicate_items$t1_t2_polycor[i] = abs(polychoric(data.frame(duplicate_items_data_t1[,c(duplicate_items$item1_ID[i])],
duplicate_items_data_t2[,c(duplicate_items$item2_ID[i])]))$rho[2])
duplicate_items$t2_t1_polycor[i] = abs(polychoric(data.frame(duplicate_items_data_t2[,c(duplicate_items$item1_ID[i])],
duplicate_items_data_t1[,c(duplicate_items$item2_ID[i])]))$rho[2])
}
duplicate_itemssummary(duplicate_items$t1_t1_polycor) Min. 1st Qu. Median Mean 3rd Qu. Max.
0.284 0.355 0.583 0.567 0.742 0.822 summary(duplicate_items$t2_t2_polycor) Min. 1st Qu. Median Mean 3rd Qu. Max.
0.101 0.446 0.612 0.580 0.700 0.904 summary(duplicate_items$t1_t2_polycor) Min. 1st Qu. Median Mean 3rd Qu. Max.
0.0128 0.3570 0.5710 0.5160 0.6870 0.8380 summary(duplicate_items$t2_t1_polycor) Min. 1st Qu. Median Mean 3rd Qu. Max.
0.0631 0.4490 0.5350 0.5010 0.5830 0.8190 Comparison to prior literature
Read in and process bootstrapped results.
process_boot_df = function(df){
df = df %>%
drop_na() %>%
mutate(dv = as.character(dv),
icc = as.numeric(as.character(icc)),
spearman = as.numeric(as.character(spearman)),
pearson = as.numeric(as.character(pearson)),
eta_sq = as.numeric(as.character(eta_sq)),
sem = as.numeric(as.character(sem)),
partial_eta_sq = as.numeric(as.character(partial_eta_sq)),
omega_sq = as.numeric(as.character(omega_sq)),
var_subs = as.numeric(as.character(var_subs)),
var_ind = as.numeric(as.character(var_ind)),
var_resid = as.numeric(as.character(var_resid)),
F_time = as.numeric(as.character(F_time)),
p_time = as.numeric(as.character(p_time)),
df_time = as.numeric(as.character(df_time)),
df_resid = as.numeric(as.character(df_resid)))
return(df)}
boot_df <- read.csv(gzfile(paste0(retest_data_path,'bootstrap_merged.csv.gz')))Warning in scan(file = file, what = what, sep = sep, quote = quote, dec =
dec, : embedded nul(s) found in inputboot_df = process_boot_df(boot_df)
boot_df = boot_df[boot_df$dv %in% retest_report_vars,]
# Check if you have all variables bootstrapped
# retest_report_vars[which(retest_report_vars %in% boot_df$dv==FALSE)]
# Boot df contains hddm parameters fit on the full sample in the t1 data
# refits_bootstrap_merged.csv.gz contains bootstrapped reliabilities
refit_boot_df = read.csv(gzfile(paste0(retest_data_path,'refits_bootstrap_merged.csv.gz')))Warning in scan(file = file, what = what, sep = sep, quote = quote, dec =
dec, : embedded nul(s) found in inputrefit_boot_df = process_boot_df(refit_boot_df)
fullfit_boot_df = boot_df[as.character(boot_df$dv) %in% unique(as.character(refit_boot_df$dv)),]
boot_df = boot_df[!as.character(boot_df$dv) %in% unique(as.character(refit_boot_df$dv)),]
boot_df = rbind(boot_df, refit_boot_df)
rm(refit_boot_df)Summarize bootstrapped results and merge to lit review data
var_boot_df = boot_df %>%
group_by(dv) %>%
summarise(mean_icc = mean(icc),
mean_pearson = mean(pearson))
rel_comp = lit_review %>%
left_join(var_boot_df, by = 'dv')Warning: Column `dv` joining factor and character vector, coercing into
character vectorHere’s what our data looks like: (583 data points for 171 measures)
rel_compLit review results
Distribution of reliabilities, sample sizes and delays
rel_comp %>%
select(dv, task, retest_reliability, sample_size, days) %>%
filter(days < 3600) %>%
gather(key, value, -dv, -task) %>%
ggplot(aes(value, fill=task))+
geom_density(alpha=0.5, position='identity')+
facet_wrap(~key, scales='free')+
theme(legend.title = element_blank())
summary(rel_comp$sample_size[rel_comp$task == "survey"]) Min. 1st Qu. Median Mean 3rd Qu. Max.
11 38 73 126 153 791 summary(rel_comp$sample_size[rel_comp$task == "task"]) Min. 1st Qu. Median Mean 3rd Qu. Max.
10.0 30.0 47.0 78.6 81.0 1120.0 The literature has smaller sized samples for task measures compared to survey measures that report retest reliability.
summary(lm(sample_size ~ task,rel_comp))
Call:
lm(formula = sample_size ~ task, data = rel_comp)
Residuals:
Min 1Q Median 3Q Max
-115.5 -57.6 -33.6 7.4 1043.4
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) 126.50 8.47 14.94 < 2e-16 ***
tasktask -47.89 10.79 -4.44 0.000011 ***
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
Residual standard error: 127 on 581 degrees of freedom
Multiple R-squared: 0.0328, Adjusted R-squared: 0.0311
F-statistic: 19.7 on 1 and 581 DF, p-value: 0.0000108What predicts retest reliability in the literature? Task, sample size, days
mod1 = lmer(retest_reliability ~ task + (1|dv), rel_comp)
mod2 = lmer(retest_reliability ~ task + sample_size + (1|dv), rel_comp)
anova(mod1, mod2)refitting model(s) with ML (instead of REML)mod3 = lmer(retest_reliability ~ task + sample_size + days+ (1|dv), rel_comp)
anova(mod2, mod3)refitting model(s) with ML (instead of REML)summary(mod2)Linear mixed model fit by REML ['lmerMod']
Formula: retest_reliability ~ task + sample_size + (1 | dv)
Data: rel_comp
REML criterion at convergence: -377.2
Scaled residuals:
Min 1Q Median 3Q Max
-3.190 -0.488 0.127 0.550 2.575
Random effects:
Groups Name Variance Std.Dev.
dv (Intercept) 0.0174 0.132
Residual 0.0206 0.143
Number of obs: 583, groups: dv, 171
Fixed effects:
Estimate Std. Error t value
(Intercept) 0.7235198 0.0213512 33.9
tasktask -0.1405121 0.0258197 -5.4
sample_size -0.0001256 0.0000543 -2.3
Correlation of Fixed Effects:
(Intr) tsktsk
tasktask -0.780
sample_size -0.319 0.117Tasks have significantly lower reliability and reliability decreases with increasing sample size.
rel_comp %>%
ggplot(aes(sample_size, retest_reliability, color=task))+
geom_smooth(method='lm')+
geom_point(alpha = 0.2)+
theme(legend.title = element_blank())+
xlab("Sample Size")+
ylab("Retest reliability")
I used to compare effect sizes of effect of task on reliability estimates in literature vs our results (just looking at the variables you find in the literature) but removed this analysis because
1. the estimates in the literature are not all the same statistic 2. from our data I was only using ICC’s 3. I was sampling from our bootstrapped results as many datapoints for each measure as there are in the literature but that didn’t seem like the best way to make use of our data to make it comparable to the literature.
Despite these problems the main result was that the effect size was much larger in our dataset compared to the literature but given the problems I think the sampling analyses below are more informative than this.
We also checked whether our results diverge most from studies with smaller sample sizes. Square difference between our mean estimate and the reliability from the literature decreases exponentially with sample size. The smaller the sample size in the literature the more the reliability estimate differs from our results. But this was a weak result because most of the studies in the literature have smaller sample sizes and you see both small and large deviations for these studies (these were not significant either).
Direct relation to our results
Correlation between our mean estimates from bootstrapped samples and the literature review for task variables
n_df = rel_comp %>%
group_by(dv) %>%
tally()
lit_emp_cor = function(){
boot_comp = data.frame()
for(i in 1:length(unique(rel_comp$dv)) ){
cur_dv = unique(rel_comp$dv)[i]
n = n_df$n[n_df$dv == cur_dv]
sample_df = boot_df %>% filter(dv == cur_dv)
tmp = sample_n(sample_df, n)
boot_comp = rbind(boot_comp, tmp)
}
rm(cur_dv, n, sample_df, tmp)
#check if cbind is ok
# sum(boot_comp$dv == rel_comp$dv)
#cbinding pearson because that is the most common metric in the lit
rel_comp = cbind(rel_comp, boot_comp$pearson)
#rename new column
names(rel_comp)[which(names(rel_comp) == "boot_comp$pearson")] = "pearson"
out = data.frame(task = NA, survey = NA)
out$task = with(rel_comp %>% filter(task == "task"), cor(pearson, retest_reliability))
out$survey = with(rel_comp %>% filter(task == "survey"), cor(pearson, retest_reliability))
rel_comp = rel_comp[,-16]
return(out)
}
lit_emp_cor_out = plyr::rdply(100, lit_emp_cor)
write.csv(lit_emp_cor_out,'/Users/zeynepenkavi/Dropbox/PoldrackLab/SRO_Retest_Analyses/output/tables/lit_emp_cor_out.csv')
summary(lit_emp_cor_out) .n task survey
Min. : 1.0 Min. :0.235 Min. :0.0381
1st Qu.: 25.8 1st Qu.:0.263 1st Qu.:0.1162
Median : 50.5 Median :0.274 Median :0.1392
Mean : 50.5 Mean :0.274 Mean :0.1381
3rd Qu.: 75.2 3rd Qu.:0.285 3rd Qu.:0.1582
Max. :100.0 Max. :0.323 Max. :0.2478 Noise ceiling
Model comparisons building model to predict the reliabilities in the literature from a sample from our results versus a sample form the literature.
sample one row per measure out of lit review r^2 of retest_reliability ~ sampled_reliability vs. r^2 of retest_reliability ~ mean_icc
coef of retest_reliability ~ sampled_reliability vs. coef of retest_reliability ~ mean_icc
comp_lit_pred <- function(df){
sample_from_dv <- function(df){
if(nrow(df)>1){
row_num = sample(1:nrow(df),1)
sample_row = df[row_num,]
df = df[-row_num,]
df$lit_predictor = sample_row$retest_reliability
}
return(df)
}
sampled_df = df %>%
group_by(dv) %>%
do(sample_from_dv(.))
mod_lit = lm(retest_reliability ~ lit_predictor+scale(sample_size)+task, data=sampled_df)
mod_boot = lm(retest_reliability ~ mean_pearson+scale(sample_size)+task, data=sampled_df)
out = data.frame(r2_lit = summary(mod_lit)$r.squared,
r2_boot = summary(mod_boot)$r.squared,
m_lit = coef(summary(mod_lit))["lit_predictor","Estimate"],
m_boot = coef(summary(mod_boot))["mean_pearson","Estimate"])
return(out)
}
comp_lit_pred_out = plyr::rdply(1000, comp_lit_pred(rel_comp))tmp = comp_lit_pred_out %>%
select(-.n) %>%
gather(key, value) %>%
separate(key, c("stat", "sample"), sep = "_")
tmp$stat = as.factor(tmp$stat)
levels(tmp$stat) <- c("coefficient", expression(r^"2"))
tmp %>%
ggplot(aes(value, fill=sample))+
geom_density(alpha = 0.5, position='identity', color=NA)+
facet_grid(.~stat, scales='free', labeller = label_parsed)+
scale_fill_discrete(breaks=c("boot","lit"),
labels=c("Empirical", "Literature"),
name="Prediction")+
xlab('')+
ylab('')
ggsave('Lit_Noise_Ceiling.jpg', device = "jpeg", path = "/Users/zeynepenkavi/Dropbox/PoldrackLab/SRO_Retest_Analyses/output/figures/", width = 10, height = 4, units = "in", limitsize = FALSE, dpi = 100)with(comp_lit_pred_out, t.test(r2_lit, r2_boot, paired=T))
Paired t-test
data: r2_lit and r2_boot
t = 29, df = 1000, p-value <2e-16
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
0.02467 0.02821
sample estimates:
mean of the differences
0.02644 with(comp_lit_pred_out, t.test(m_lit, m_boot))
Welch Two Sample t-test
data: m_lit and m_boot
t = 18, df = 1300, p-value <2e-16
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
0.02791 0.03472
sample estimates:
mean of x mean of y
0.3334 0.3020 Relationship between reliability metrics (point estimates)
Based on Weir (2005) ICC(3,k) does not take in to account within subject differences between two time points (i.e. the fixed effect of time/systematic error). Thus, it is well approximated by Pearson’s r and subject to similar criticisms. Weir (2005) suggests reporting at least this systematic error effect size if one chooses to report with ICC(3,k). Based on his conclusions here I report:
- ICC(3,k): As Dave clarified this ranges from 1 to -1/(number of repeated measures -1) so in our case this range would be [-1, 1]; larger values would mean that the two scores of a subject for a given measure are more similar to each other than they are to scores of other people
- “ICC is reflective of the ability of a test to differentiate between different individuals” - partial \(\eta^2\) for time (\(SS_{time}/SS_{within}\)): effect size of time
- SEM (\(\sqrt(MS_{error})\)): standard error of measurement; the smaller the better. It “quantifies the precision of individual scores on a test” and is not dependent on the sample in the way ICC is since it doesn’t depend on between subject reliability (at least in this formulation) but is unit-dependent.
We calculated 74 measures for surveys and 372 measures for cognitive tasks.
Though we are primarily reporting ICC’s as our metric of reliability the results don’t change depending on the metric chosen. Here we plot point estimates of three different reliability metrics against each other (ICC, Pearson, Spearman). The bootstrapped version is essentially the same but the plots are busier due to more datapoints.
table(rel_df$task)
survey task
74 372 p1 = rel_df %>%
ggplot(aes(spearman, icc, col=task))+
geom_point()+
theme_bw()+
theme(legend.title = element_blank(),
legend.position = "none")+
geom_abline(intercept = 0, slope=1)
p2 = rel_df %>%
ggplot(aes(pearson, icc, col=task))+
geom_point()+
theme_bw()+
theme(legend.title = element_blank(),
legend.position = "none")+
geom_abline(intercept = 0, slope=1)
p3 = rel_df %>%
ggplot(aes(pearson, spearman, col=task))+
geom_point()+
theme_bw()+
theme(legend.title = element_blank(),
legend.position = "none")+
geom_abline(intercept = 0, slope=1)
grid.arrange(p1, p2, p3, nrow=1)
ggsave('Metric_Scatterplots.jpg', plot = grid.arrange(p1, p2, p3, nrow=1), device = "jpeg", path = "/Users/zeynepenkavi/Dropbox/PoldrackLab/SRO_Retest_Analyses/output/figures/", width = 12, height = 4, units = "in", limitsize = FALSE, dpi = 100)As the scatter plots depict the correlations between different types reliability metrics were very high.
cor(rel_df[,c('spearman', 'icc', 'pearson')]) spearman icc pearson
spearman 1.0000 0.9324 0.9577
icc 0.9324 1.0000 0.9800
pearson 0.9577 0.9800 1.0000Note: Some variables have <0 ICC’s. This would be the case if the \(MS_{error}\)>\(MS_{between}\). Data for these variables have no relationship between the two time points.
Summary of all measure reliabilities
Summarized bootstrapped reliabilities
boot_df %>%
group_by(dv) %>%
summarise(icc_median = quantile(icc, probs = 0.5),
icc_2.5 = quantile(icc, probs = 0.025),
icc_97.5 = quantile(icc, probs = 0.975),
spearman_median = quantile(spearman, probs = 0.5),
spearman_2.5 = quantile(spearman, probs = 0.025),
spearman_97.5 = quantile(spearman, probs = 0.975)) %>%
datatable() %>%
formatRound(columns=c('icc_median', 'icc_2.5', 'icc_97.5', 'spearman_median', 'spearman_2.5', 'spearman_97.5'), digits=3)# Df wrangling for plotting
tmp = measure_labels %>%
mutate(dv = as.character(dv)) %>%
left_join(boot_df[,c("dv", "icc", "spearman")], by = 'dv')
tmp = tmp %>%
separate(dv, c("task_group", "var"), sep="\\.",remove=FALSE,extra="merge") %>%
mutate(task_group = factor(task_group, levels = task_group[order(task)])) %>%
separate(var, c("var"), sep="\\.",remove=TRUE,extra="drop") %>%
mutate(task_group = gsub("_", " ", task_group),
var = gsub("_", " ", var)) %>%
arrange(task_group, var)
tmp = tmp %>%
left_join(rel_df[,c("dv", "icc")], by = "dv") %>%
rename(icc = icc.x, point_est = icc.y)
#Manual correction
tmp = tmp %>%
mutate(task = ifelse(task_group == 'holt laury survey', "task", as.character(task))) %>%
mutate(task_group = ifelse(task_group == "psychological refractory period two choices", "psychological refractory period", ifelse(task_group == "angling risk task always sunny", "angling risk task",task_group))) %>%
mutate(task_group = gsub("survey", "", task_group))Variable level summary of bootstrapped reliabilities.
p4 <- tmp %>%
filter(task == 'task',
raw_fit == 'raw') %>%
ggplot(aes(y = var, x = icc)) +
geom_point(color = '#00BFC4')+
geom_point(aes(y = var, x = point_est), color = "black")+
facet_grid(task_group~., switch = "y", scales = "free_y", space = "free_y", labeller = label_wrap_gen(width=20)) +
theme(panel.spacing = unit(0.75, "lines"),
strip.placement = "outside",
strip.text.y = element_text(angle=180, size=36),
axis.text.y = element_text(size=20),
panel.background = element_rect(fill = NA),
panel.grid.major = element_line(colour = "grey80")) +
xlab("")+
ylab("")+
scale_x_continuous(limits = c(-0.25,1), breaks=c(-0.25, 0, 0.25, 0.5, 0.75, 1))+
scale_y_discrete(labels = function(x) str_wrap(x, width = 10))+
geom_vline(xintercept = 0, color = "red", size = 1)
p5 <- tmp %>%
filter(task == 'survey') %>%
ggplot(aes(y = var, x = icc)) +
geom_point(color = '#F8766D')+
geom_point(aes(y = var, x = point_est), color = "black")+
facet_grid(task_group~., switch = "y", scales = "free_y", space = "free_y", labeller = label_wrap_gen(width=20)) +
theme(panel.spacing = unit(0.75, "lines"),
strip.placement = "outside",
strip.text.y = element_text(angle=180, size=36),
axis.text.y = element_text(size=20),
panel.background = element_rect(fill = NA),
panel.grid.major = element_line(colour = "grey80")) +
xlab("")+
ylab("")+
scale_x_continuous(limits = c(-0.25,1), breaks=c(-0.25, 0, 0.25, 0.5, 0.75, 1))+
scale_y_discrete(labels = function(x) str_wrap(x, width = 10))+
geom_vline(xintercept = 0, color = "red", size = 1)
p6 <- arrangeGrob(p4, p5,nrow=1)
ggsave('Bootstrap_Raw_Var_Plot.jpg', plot = p6, device = "jpeg", path = "/Users/zeynepenkavi/Dropbox/PoldrackLab/SRO_Retest_Analyses/output/figures/", width = 36, height = 72, units = "in", limitsize = FALSE, dpi=50)
rm(p4, p5, p6)
p4_t <- tmp %>%
filter(task == 'task') %>%
ggplot(aes(x = factor(task_group, levels=rev(unique(task_group))), y = icc)) +
geom_violin(fill='#00BFC4')+
theme_bw() +
theme(axis.text.y = element_text(size=30))+
xlab("")+
ylab("")+
scale_y_continuous(limits = c(-0.25,1), breaks=c(-0.25, 0, 0.25, 0.5, 0.75, 1))+
scale_x_discrete(labels = function(x) str_wrap(x, width = 15))+
geom_hline(yintercept = 0, color = "red", size = 1)+
coord_flip()
p5_t <- tmp %>%
filter(task == 'survey') %>%
ggplot(aes(x = factor(task_group, levels=rev(unique(task_group))), y = icc)) +
geom_violin(fill='#F8766D')+
theme_bw() +
theme(axis.text.y = element_text(size=43))+
xlab("")+
ylab("")+
scale_y_continuous(limits = c(-0.25,1), breaks=c(-0.25, 0, 0.25, 0.5, 0.75, 1))+
scale_x_discrete(labels = function(x) str_wrap(x, width = 10))+
geom_hline(yintercept = 0, color = "red", size = 1)+
coord_flip()
p6_t <- arrangeGrob(p4_t, p5_t,nrow=1)
ggsave('Bootstrap_Poster_Plot_t.jpg', plot = p6_t, device = "jpeg", path = "../output/figures/", width = 27, height = 48, units = "in", limitsize = FALSE, dpi = 100)
Example of the overlaying procedure.
p1<- tmp %>%
filter(grepl('selection_optimization', dv)) %>%
ggplot(aes(x = var, y = icc)) +
geom_violin(fill='#F8766D')+
theme_bw() +
theme(axis.text.y = element_text(size=30))+
xlab("")+
ylab("")+
scale_y_continuous(limits = c(-0.25,1), breaks=c(-0.25, 0, 0.25, 0.5, 0.75, 1))+
scale_x_discrete(labels = function(x) str_wrap(x, width = 15))+
geom_hline(yintercept = 0, color = "red", size = 1)+
coord_flip()
p2<- tmp %>%
filter(grepl('selection_optimization', dv)) %>%
ggplot(aes(x = factor(task_group, levels=rev(unique(task_group))), y = icc, group=dv)) +
geom_violin(fill='#F8766D', position = 'identity')+
theme_bw() +
theme(axis.text.y = element_text(size=30))+
xlab("")+
ylab("")+
scale_y_continuous(limits = c(-0.25,1), breaks=c(-0.25, 0, 0.25, 0.5, 0.75, 1))+
scale_x_discrete(labels = function(x) str_wrap(x, width = 15))+
geom_hline(yintercept = 0, color = "red", size = 1)+
coord_flip()
p3<- tmp %>%
filter(grepl('selection_optimization', dv)) %>%
ggplot(aes(x = factor(task_group, levels=rev(unique(task_group))), y = icc)) +
geom_violin(fill='#F8766D', position = 'identity')+
theme_bw() +
theme(axis.text.y = element_text(size=30))+
xlab("")+
ylab("")+
scale_y_continuous(limits = c(-0.25,1), breaks=c(-0.25, 0, 0.25, 0.5, 0.75, 1))+
scale_x_discrete(labels = function(x) str_wrap(x, width = 15))+
geom_hline(yintercept = 0, color = "red", size = 1)+
coord_flip()
p4 <- arrangeGrob(p1,p2, p3,nrow=1)
ggsave('Bootstrap_Example_Plot_t.jpg', plot = p4, device = "jpeg", path = "../output/figures/", width = 41, height = 5, units = "in", limitsize = FALSE, dpi = 100)
rm(p1, p2, p3, p4)Survey vs Tasks
Comparison of survey measures to cognitive task measures in the bootstrapped results. Multilevel model with random intercepts for each measure and fixed effect of survey versus cognitive measure.
boot_df = boot_df %>%
mutate(task = ifelse(grepl("survey",dv), "survey","task"),
task = ifelse(grepl("holt",dv), "task", task))
boot_df %>%
group_by(task) %>%
summarise(icc_median = quantile(icc, probs = 0.5),
icc_2.5 = quantile(icc, probs = 0.025),
icc_97.5 = quantile(icc, probs = 0.975),
spearman_median = quantile(spearman, probs = 0.5),
spearman_2.5 = quantile(spearman, probs = 0.025),
spearman_97.5 = quantile(spearman, probs = 0.975),
num_vars = n()/1000) %>%
datatable() %>%
formatRound(columns=c('icc_median', 'icc_2.5', 'icc_97.5', 'spearman_median', 'spearman_2.5', 'spearman_97.5'), digits=3)summary(lmerTest::lmer(icc ~ task + (1|dv), boot_df))Linear mixed model fit by REML t-tests use Satterthwaite approximations
to degrees of freedom [lmerMod]
Formula: icc ~ task + (1 | dv)
Data: boot_df
REML criterion at convergence: -942508
Scaled residuals:
Min 1Q Median 3Q Max
-13.201 -0.401 0.024 0.459 7.051
Random effects:
Groups Name Variance Std.Dev.
dv (Intercept) 0.05354 0.2314
Residual 0.00701 0.0837
Number of obs: 446000, groups: dv, 446
Fixed effects:
Estimate Std. Error df t value Pr(>|t|)
(Intercept) 0.8725 0.0269 449.0000 32.4 <2e-16 ***
tasktask -0.3903 0.0295 449.0000 -13.2 <2e-16 ***
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
Correlation of Fixed Effects:
(Intr)
tasktask -0.913Variance breakdown
The quantiative explanation for the difference in reliability estimates between surveys and tasks, as recently detailed by Hedge et al. (2017), lies in the difference in sources of variance between these measures. Specifically, the ICC is calculated as the ratio of variance between subjects variance to all sources of variance. Thus, measures with high between subjects variance would have high test-retest reliability. Intuitively, measures with high between subjects variance are also better suited for individual difference analyses as they would capture the differences between the subjects in a sample.
Here we first plot the percentage of variance explained by the three sources of variance for the point estimates of measure reliabilities. The plot only includes raw measures (no DDM parameters) and the measures are ranked by percentage of between subject variability for each task/survey (i.e. the best to worst individual difference measure for each task/survey). Then we compare statistically whether the percentage of variance explained by these sources differ between tasks and surveys.
tmp = rel_df %>%
mutate(var_subs_pct = var_subs/(var_subs+var_ind+var_resid)*100,
var_ind_pct = var_ind/(var_subs+var_ind+var_resid)*100,
var_resid_pct = var_resid/(var_subs+var_ind+var_resid)*100) %>%
select(dv, task, var_subs_pct, var_ind_pct, var_resid_pct) %>%
mutate(dv = factor(dv, levels = dv[order(task)])) %>%
separate(dv, c("task_group", "var"), sep="\\.",remove=FALSE,extra="merge") %>%
mutate(task_group = factor(task_group, levels = task_group[order(task)])) %>%
arrange(task_group, var_subs_pct) %>%
mutate(rank = row_number()) %>%
arrange(task, task_group, rank) %>%
gather(key, value, -dv, -task_group, -var, -task, -rank) %>%
ungroup()%>%
mutate(task_group = gsub("_", " ", task_group),
var = gsub("_", " ", var)) %>%
mutate(task_group = ifelse(task_group == "psychological refractory period two choices", "psychological refractory period", ifelse(task_group == "angling risk task always sunny", "angling risk task",task_group))) %>%
mutate(task_group = gsub("survey", "", task_group)) %>%
filter(task=="task",
!grepl("EZ|hddm", dv))%>%
arrange(task_group, rank)
labels = tmp %>%
distinct(dv, .keep_all=T)
p1 <- tmp %>%
ggplot(aes(x=factor(rank), y=value, fill=factor(key, levels = c("var_resid_pct", "var_ind_pct", "var_subs_pct"))))+
geom_bar(stat='identity', alpha = 0.75, color='#00BFC4')+
scale_x_discrete(breaks = labels$rank,
labels = labels$var)+
coord_flip()+
facet_grid(task_group~., switch = "y", scales = "free_y", space = "free_y") +
theme(panel.spacing = unit(0.5, "lines"),
strip.placement = "outside",
strip.text.y = element_text(angle=180),
panel.background = element_rect(fill = NA),
panel.grid.major = element_line(colour = "grey85"),
legend.position = 'bottom')+
theme(legend.title = element_blank())+
scale_fill_manual(breaks = c("var_subs_pct", "var_ind_pct", "var_resid_pct"),
labels = c("Variance between individuals",
"Variance between sessions",
"Error variance"),
values=c("grey65", "grey45", "grey25"))+
ylab("")+
xlab("")
tmp = rel_df %>%
mutate(var_subs_pct = var_subs/(var_subs+var_ind+var_resid)*100,
var_ind_pct = var_ind/(var_subs+var_ind+var_resid)*100,
var_resid_pct = var_resid/(var_subs+var_ind+var_resid)*100) %>%
select(dv, task, var_subs_pct, var_ind_pct, var_resid_pct) %>%
mutate(dv = factor(dv, levels = dv[order(task)])) %>%
separate(dv, c("task_group", "var"), sep="\\.",remove=FALSE,extra="merge") %>%
mutate(task_group = factor(task_group, levels = task_group[order(task)])) %>%
arrange(task_group, var_subs_pct) %>%
mutate(rank = row_number()) %>%
arrange(task, task_group, rank) %>%
gather(key, value, -dv, -task_group, -var, -task, -rank) %>%
ungroup()%>%
mutate(task_group = gsub("_", " ", task_group),
var = gsub("_", " ", var)) %>%
mutate(task_group = ifelse(task_group == "psychological refractory period two choices", "psychological refractory period", ifelse(task_group == "angling risk task always sunny", "angling risk task",task_group))) %>%
mutate(task_group = gsub("survey", "", task_group)) %>%
filter(task=="survey")%>%
arrange(task_group, rank)
labels = tmp %>%
distinct(dv, .keep_all=T)
p2 <- tmp %>%
ggplot(aes(x=factor(rank), y=value, fill=factor(key, levels = c("var_resid_pct", "var_ind_pct", "var_subs_pct"))))+
geom_bar(stat='identity', alpha = 0.75)+
geom_bar(stat='identity', color='#F8766D', show.legend=FALSE)+
scale_x_discrete(breaks = labels$rank,
labels = labels$var)+
coord_flip()+
facet_grid(task_group~., switch = "y", scales = "free_y", space = "free_y") +
theme(panel.spacing = unit(0.5, "lines"),
strip.placement = "outside",
strip.text.y = element_text(angle=180),
panel.background = element_rect(fill = NA),
panel.grid.major = element_line(colour = "grey85"),
legend.position = 'bottom')+
theme(legend.title = element_blank())+
scale_fill_manual(breaks = c("var_subs_pct", "var_ind_pct", "var_resid_pct"),
labels = c("Variance between individuals",
"Variance between sessions",
"Error variance"),
values=c("grey65", "grey45", "grey25"))+
ylab("")+
xlab("")
mylegend<-g_legend(p2)
p3 <- arrangeGrob(arrangeGrob(p1 +theme(legend.position="none"),
p2 + theme(legend.position="none"),
nrow=1),
mylegend, nrow=2,heights=c(10, 1))
ggsave('Variance_Breakdown_Plot.jpg', plot = p3, device = "jpeg", path = "/Users/zeynepenkavi/Dropbox/PoldrackLab/SRO_Retest_Analyses/output/figures", width = 24, height = 20, units = "in", dpi = 100)
rm(tmp, labels, p1, p2 , p3)
Comparing types of variance for survey vs task measures: Survey measures have higher between subject variability
Note: This analysis includes DDM variables too.
Running separate models for different sources of variance because interactive model with variance type*task seemed too complicated.
First we find that task measures have a smaller percentage of their overall variance explained by variability between subjects compared to survey measures.
tmp = boot_df %>%
mutate(var_subs_pct = var_subs/(var_subs+var_ind+var_resid)*100,
var_ind_pct = var_ind/(var_subs+var_ind+var_resid)*100,
var_resid_pct = var_resid/(var_subs+var_ind+var_resid)*100) %>%
select(dv, task, var_subs_pct, var_ind_pct, var_resid_pct)
summary(lmerTest::lmer(var_subs_pct~task+(1|dv),tmp%>%select(-var_ind_pct,-var_resid_pct)))Linear mixed model fit by REML t-tests use Satterthwaite approximations
to degrees of freedom [lmerMod]
Formula: var_subs_pct ~ task + (1 | dv)
Data: tmp %>% select(-var_ind_pct, -var_resid_pct)
REML criterion at convergence: 3304624
Scaled residuals:
Min 1Q Median 3Q Max
-4.316 -0.621 0.103 0.647 5.381
Random effects:
Groups Name Variance Std.Dev.
dv (Intercept) 188 13.7
Residual 96 9.8
Number of obs: 446000, groups: dv, 446
Fixed effects:
Estimate Std. Error df t value Pr(>|t|)
(Intercept) 80.33 1.60 443.00 50.4 <2e-16 ***
tasktask -28.81 1.75 443.00 -16.5 <2e-16 ***
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
Correlation of Fixed Effects:
(Intr)
tasktask -0.913We also find that a significantly larger percentage of their variance is explained by between session variability. Larger between session variability suggests systematic differences between the two sessions. Such systematic effects can be due to e.g. learning effects as explored later.
summary(lmerTest::lmer(var_ind_pct~task+(1|dv),tmp%>%select(-var_subs_pct,-var_resid_pct)))Linear mixed model fit by REML t-tests use Satterthwaite approximations
to degrees of freedom [lmerMod]
Formula: var_ind_pct ~ task + (1 | dv)
Data: tmp %>% select(-var_subs_pct, -var_resid_pct)
REML criterion at convergence: 3611691
Scaled residuals:
Min 1Q Median 3Q Max
-4.412 -0.651 -0.139 0.584 4.736
Random effects:
Groups Name Variance Std.Dev.
dv (Intercept) 239 15.5
Residual 191 13.8
Number of obs: 446000, groups: dv, 446
Fixed effects:
Estimate Std. Error df t value Pr(>|t|)
(Intercept) 9.84 1.80 444.00 5.47 7.4e-08 ***
tasktask 14.36 1.97 444.00 7.29 1.4e-12 ***
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
Correlation of Fixed Effects:
(Intr)
tasktask -0.913summary(lmerTest::lmer(var_resid_pct~task+(1|dv),tmp%>%select(-var_subs_pct,-var_ind_pct)))Linear mixed model fit by REML t-tests use Satterthwaite approximations
to degrees of freedom [lmerMod]
Formula: var_resid_pct ~ task + (1 | dv)
Data: tmp %>% select(-var_subs_pct, -var_ind_pct)
REML criterion at convergence: 2777592
Scaled residuals:
Min 1Q Median 3Q Max
-6.323 -0.475 0.018 0.530 6.343
Random effects:
Groups Name Variance Std.Dev.
dv (Intercept) 75.2 8.67
Residual 29.4 5.43
Number of obs: 446000, groups: dv, 446
Fixed effects:
Estimate Std. Error df t value Pr(>|t|)
(Intercept) 9.83 1.01 446.00 9.75 <2e-16 ***
tasktask 14.45 1.10 446.00 13.09 <2e-16 ***
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
Correlation of Fixed Effects:
(Intr)
tasktask -0.913tmp_save = tmp%>%
gather(key, value, -dv, -task) %>%
group_by(task, key) %>%
summarise(median = median(value),
sd = sd(value)) %>%
mutate(key = ifelse(key == 'var_ind_pct', 'Between session variance', ifelse(key == 'var_subs_pct', 'Between subjects variance', ifelse(key == 'var_resid_pct', 'Residual variance',NA)))) %>%
rename(Median = median, SD = sd) %>%
arrange(task, key)
tmp_savesjt.df(tmp_save %>% mutate_if(is.numeric, funs(round(., 3))), describe=F, hide.progress = TRUE, show.rownames = FALSE, file = "/Users/zeynepenkavi/Dropbox/PoldrackLab/SRO_Retest_Analyses/output/tables/var_breakdown_summary.doc")| task | key | Median | SD |
|---|---|---|---|
| survey | Between session variance | 5.384 | 11.353 |
| survey | Between subjects variance | 83.355 | 11.686 |
| survey | Residual variance | 8.976 | 4.325 |
| task | Between session variance | 18.077 | 22.114 |
| task | Between subjects variance | 52.548 | 17.681 |
| task | Residual variance | 22.818 | 11.012 |
Summarizing for clearer presentation. This graph is currently using the bootstrapped reliabilities and is therefore messier than if just using the point estimates.
tmp %>%
gather(key, value, -dv, -task) %>%
group_by(task, key) %>%
summarise(mean_pct = mean(value),
sd_pct = sd(value),
n = n()) %>%
mutate(cvl = qt(0.025, n-1),
cvu = qt(0.975, n-1),
cil = mean_pct+(sd_pct*cvl)/sqrt(n),
ciu = mean_pct+(sd_pct*cvu)/sqrt(n),
sem_pct = sd_pct/sqrt(n)) %>%
ggplot(aes(factor(key, levels = c("var_subs_pct", "var_ind_pct", "var_resid_pct"),
labels = c("Variance between individuals",
"Variance between sessions",
"Error variance")), mean_pct, color=task))+
geom_point(position=position_dodge(width = 0.25), size = 4)+
# geom_errorbar(aes(ymin=mean_pct-sd_pct, ymax=mean_pct+sd_pct), position=position_dodge(width = 0.25), width=0, size=2)+
geom_errorbar(aes(ymin=cil, ymax=ciu), position=position_dodge(width = 0.25), width=0.25, size=2)+
theme_bw()+
xlab('')+
ylab('Percent')+
theme(legend.title = element_blank(),
legend.text = element_text(size=14),
legend.position = 'bottom',
axis.text = element_text(size=12),
axis.title.y = element_text(size=12))+
scale_x_discrete(labels = function(x) str_wrap(x, width = 15))
ggsave('Variance_Breakdown_DotPlot.jpg', device = "jpeg", path = "/Users/zeynepenkavi/Dropbox/PoldrackLab/SRO_Retest_Analyses/output/figures", width = 6, height = 5, units = "in", dpi = 100)Systematic effects between time points
Anova time effects
The type of ICC we have chosen does not take within subject (between session/systematic) variance in to account. This is why Weir recommends checking whether there is a significant change based on time and examining the SEMs. These systematic effects could be meaningful and important to account for for some measures (e.g. task measures that show learning effects).
Had we chosen another kind of ICC taking this source of variance into account (e.g. 2,1 or 2,k) they could have suggested that tasks have lower reliability.
Doing a simple t-test on the difference score alone would not be a very rigourous way of testing whether any change is meaningful because two distributions from both time points with error would be compared to each other. Fortunately there are ways to take the error for both measurements in to account.
To check whether a measure shows systematic differences between the two time points in a meaningful number of bootstrapped samples we can: check if the effect of time is significant in each bootstrapped sample and filter variables that have more than 5% of the boostrapped samples showing significant time effects.
Another way might be to compute confidence intervals using SEMs as described in the second half of Weir (2005) and check what percent of participants have scores that fall out of this range. I haven’t pursued this for now.
Here we ask: Which variables have significant time effects in more than 5% of the bootstrapped samples?
23/74 survey measures 133/372 task measures
boot_df %>%
select(dv, p_time, task) %>%
mutate(time_effect_sig = ifelse(p_time<0.05,1,0)) %>%
group_by(dv)%>%
summarise(pct_sig_time_effect = sum(time_effect_sig)/10,
task = unique(task))%>%
filter(pct_sig_time_effect>5) %>%
arrange(task,-pct_sig_time_effect) %>%
ungroup()%>%
group_by(task) %>%
summarise(count=n())Are these significantly different between surveys and tasks? No.
chisq.test(matrix(data = c(23,74-23,133, 372-133), nrow=2))
Pearson's Chi-squared test with Yates' continuity correction
data: matrix(data = c(23, 74 - 23, 133, 372 - 133), nrow = 2)
X-squared = 0.4, df = 1, p-value = 0.5Are they predominantly in one or the other direction and does that differ depending on survey vs task?
boot_df %>%
select(dv, p_time, task, pearson) %>%
mutate(time_effect_sig = ifelse(p_time<0.05,1,0)) %>%
group_by(dv)%>%
summarise(pct_sig_time_effect = sum(time_effect_sig)/10,
task = unique(task),
mean_pearson = mean(pearson))%>%
filter(pct_sig_time_effect>5) %>%
arrange(task,-pct_sig_time_effect) %>%
arrange(mean_pearson) SEM CI calculation
Step 1: T = Grand mean + ICC * (Subject score - Grand mean) Step 2: SEP = SD of both measurements * sqrt(1-ICC^2)
Here I calculate the proportion of subjects that move more than one standard error of prediction (SEP) in t2 compared to t1 for each measure.
This is odd to think about because the larger the ICC of a measure the smaller the SEP. So very small differences between the two time points can be categorized as ‘meaningful’ based on the tiny SEP.
What you are interested in is not necessarily whether individuals change at all between the two time points though. You want to know if this change is systematic in one direction.
Here I calculate the proportion of people showing ‘meaningful’ changes in one direction or the other. To integrate both of these direction I subtracted one from the other and filtered the variables that have more than 5% of the participants showing meaningful change in one direction over the other (so if a variable has 10 participants showing difference in one and 10 in the other direction this would cancel out but a variables with 15 people showing a positive and 5 negative change would remain).
get_ind_ci = function(dv_var){
matched = match_t1_t2(dv_var)
grand_mean = mean(matched$score)
grand_sd = sd(matched$score)
dv_icc = get_icc(dv_var)
sep = grand_sd * sqrt(1-(dv_icc^2))
matched = matched %>%
spread(time, score) %>%
rename("t1"="1", "t2"="2") %>%
mutate(true_score = grand_mean+(dv_icc*(t1-grand_mean)),
ci_up = true_score+(1.96*sep),
ci_low = true_score-(1.96*sep),
t2_above_ci = ifelse(t2>ci_up,1,0),
t2_below_ci = ifelse(t2<ci_low,1, 0))
return(matched)
}
get_prop_out_ci = function(dv_var){
get_ind_ci(dv_var) %>%
summarise(prop_above_ci = sum(t2_above_ci)/n(),
prop_below_ci = sum(t2_below_ci/n()))
}
#Create df of point estimate reliabilities
ind_ci_df <- data.frame(prop_above_ci = rep(NA, length(numeric_cols)),
prop_below_ci = rep(NA, length(numeric_cols)))
row.names(ind_ci_df) <- numeric_cols
for(i in 1:length(numeric_cols)){
ind_ci_df[numeric_cols[i], 'prop_above_ci'] <- get_prop_out_ci(numeric_cols[i])$prop_above_ci
ind_ci_df[numeric_cols[i], 'prop_below_ci'] <- get_prop_out_ci(numeric_cols[i])$prop_below_ci
}
ind_ci_df$dv = row.names(ind_ci_df)
row.names(ind_ci_df) = seq(1:nrow(ind_ci_df))
ind_ci_df$task = 'task'
ind_ci_df[grep('survey', ind_ci_df$dv), 'task'] = 'survey'
ind_ci_df[grep('holt', ind_ci_df$dv), 'task'] = "task"
ind_ci_df = ind_ci_df %>%
select(dv, task, prop_above_ci, prop_below_ci)
mean_diff_df = ind_ci_df %>%
mutate(prop_one_direction = prop_above_ci-prop_below_ci) %>%
filter(abs(prop_one_direction)>0.05) %>%
arrange(-abs(prop_one_direction))
mean_diff_df# write.csv(mean_diff_df, '/Users/zeynepenkavi/Dropbox/PoldrackLab/SRO_Retest_Analyses/output/tables/mean_diff_df.csv')This doesn’t make it easier to interpret whether it is a performance improvement or decline. In fact ‘performance’ isn’t even necessarily the correct term here. Using the valence assignments of the measures look at whether it translates to an increase or decrease in self-regulation. Of the 66 this makes sense for 54 variables.
mean_diff_df = read.csv('/Users/zeynepenkavi/Dropbox/PoldrackLab/SRO_Retest_Analyses/output/tables/mean_diff_df.csv')
table(as.character(mean_diff_df$valence), useNA = "ifany")
Neg Pos <NA>
24 30 12 mean_diff_df %>%
mutate(sc_increase = ifelse(prop_one_direction>0&valence=="Pos",1,ifelse(prop_one_direction<0&valence=="Neg",1,0)),
sc_decrease = ifelse(prop_one_direction>0&valence=="Neg",1,ifelse(prop_one_direction<0&valence=="Pos",1,0))) %>%
summarise(sum_sc_increase = sum(sc_increase,na.rm=T),
sum_sc_decrease = sum(sc_decrease,na.rm=T))Variables that show more than 5% difference in a direction do not depend on whether they translate to a self control increase or decrease.
chisq.test(matrix(data = c(23,54-23,31,54-31), nrow=2))
Pearson's Chi-squared test with Yates' continuity correction
data: matrix(data = c(23, 54 - 23, 31, 54 - 31), nrow = 2)
X-squared = 1.8, df = 1, p-value = 0.2Do they differ depending on whether they are task or survey variables? No.
chisq.test(matrix(data = c(5,74-5,61,372-61), nrow=2))
Pearson's Chi-squared test with Yates' continuity correction
data: matrix(data = c(5, 74 - 5, 61, 372 - 61), nrow = 2)
X-squared = 3.8, df = 1, p-value = 0.05rm(mean_diff_df)Task Reliabilities
Here we summarize the results on a task level to make it more digestable and easier to make contact with the literature.
We reduce the list of task measures to a list of one per task by averaging only the raw measures from all the trials in a task. We chose to reduce the information in this manner to avoid any bias stemming from differential amount of interest and procedures applied to certain tasks over others (e.g. a task can have over 10 measures because it has multiple conditions and we have chosen to fit DDM’s for specific conditions while another might only have 2 due to our relative inexperience and lack of interest in it). We check whether the number of trials in a task has a significant effect on these average reliabilities of raw measures as well.
We filter out the DDM parameters and measures for specific contrasts. Note that this does leave some tasks with measures that are model fits and/or for specific conditions (because at least the current datasets do not include measures that are based on all the trials and are raw though I could dive in to variables_exhaustive for such measures. For example the average relialibility for Kirby is based on three discount rates for specific conditions.). Here’s the order of tasks by mean reliability sorted for ICC and then Spearman’s \(\rho\).
tmp = measure_labels %>%
mutate(dv = as.character(dv)) %>%
filter(task == 'task') %>%
left_join(boot_df[,c("dv", "icc", "spearman")], by = 'dv') %>%
filter(overall_difference != 'difference' & raw_fit %in% c('EZ', 'hddm') == FALSE)%>%
separate(dv, c('task_name', 'extra_1', 'extra_2'), sep = '\\.',remove=FALSE) %>%
select(-extra_1, -extra_2) %>%
group_by(task_name) %>%
summarise(median_icc = median(icc),
median_spearman = median(spearman),
icc_2.5 = quantile(icc, probs = 0.025),
icc_97.5 = quantile(icc, probs = 0.975),
min_spearman = min(spearman),
max_spearman = max(spearman),
num_measures = n()/1000,
mean_num_trials = round(mean(num_all_trials)))%>%
arrange(-median_icc, -median_spearman)
tmp %>%
datatable() %>%
formatRound(columns=c('median_spearman', 'median_icc',
'min_spearman', 'icc_2.5',
'max_spearman', 'icc_97.5'), digits=3)tmp = tmp%>%
select(-median_spearman, -min_spearman, -max_spearman) %>%
mutate(task_name = gsub("_", " ", task_name),
task_name = gsub("(^|[[:space:]])([[:alpha:]])", "\\1\\U\\2", task_name, perl=TRUE))
names(tmp) = gsub("_", " ", names(tmp))
names(tmp) = gsub("(^|[[:space:]])([[:alpha:]])", "\\1\\U\\2", names(tmp), perl=TRUE)
sjt.df(tmp %>% mutate_if(is.numeric, funs(round(., 3))), describe=F, hide.progress = TRUE, show.rownames = FALSE, file = "/Users/zeynepenkavi/Dropbox/PoldrackLab/SRO_Retest_Analyses/output/tables/task_rel_table.doc")| Task Name | Median Icc | Icc 2.5 | Icc 97.5 | Num Measures | Mean Num Trials |
|---|---|---|---|---|---|
| Ravens | 0.875 | 0.845 | 0.904 | 1 | 18 |
| Adaptive N Back | 0.842 | 0.362 | 0.911 | 3 | 380 |
| Kirby | 0.835 | 0.746 | 0.897 | 12 | 14 |
| Cognitive Reflection Survey | 0.783 | 0.669 | 0.865 | 2 | 6 |
| Threebytwo | 0.772 | 0.694 | 0.845 | 2 | 383 |
| Recent Probes | 0.772 | 0.629 | 0.847 | 2 | 68 |
| Stroop | 0.76 | 0.445 | 0.844 | 7 | 67 |
| Simple Reaction Time | 0.738 | 0.668 | 0.826 | 1 | 149 |
| Psychological Refractory Period Two Choices | 0.732 | 0.499 | 0.85 | 4 | 186 |
| Choice Reaction Time | 0.727 | 0.506 | 0.848 | 2 | 150 |
| Hierbackground-color:#eaeaea;hical Rule | 0.707 | 0.592 | 0.792 | 3 | 298 |
| Attention Network Task | 0.703 | -0.513 | 0.814 | 6 | 75 |
| Simon | 0.692 | 0.398 | 0.821 | 8 | 61 |
| Directed Forgetting | 0.681 | 0.479 | 0.782 | 2 | 67 |
| Digit Span | 0.675 | 0.544 | 0.786 | 2 | 10 |
| Shape Matching | 0.673 | 0.57 | 0.763 | 2 | 272 |
| Information Sampling Task | 0.67 | 0.118 | 0.859 | 8 | 10 |
| Stop Signal | 0.661 | 0.261 | 0.815 | 11 | 304 |
| Spatial Span | 0.658 | 0.382 | 0.794 | 2 | 10 |
| Discount Titrate | 0.645 | 0.496 | 0.753 | 1 | 36 |
| Go Nogo | 0.643 | 0.256 | 0.785 | 5 | 210 |
| Keep Track | 0.641 | 0.497 | 0.719 | 1 | 9 |
| Tower Of London | 0.634 | 0.436 | 0.855 | 4 | 12 |
| Dot Pattern Expectancy | 0.599 | 0.111 | 0.798 | 11 | 61 |
| Columbia Card Task Hot | 0.597 | 0.433 | 0.857 | 5 | 24 |
| Stim Selective Stop Signal | 0.577 | 0.375 | 0.752 | 4 | 104 |
| Two Stage Decision | 0.575 | 0.294 | 0.746 | 4 | 198 |
| Local Global Letter | 0.57 | 0.285 | 0.76 | 12 | 31 |
| Dietary Decision | 0.564 | 0.088 | 0.749 | 3 | 48 |
| Columbia Card Task Cold | 0.524 | 0.248 | 0.688 | 5 | 24 |
| Holt Laury Survey | 0.524 | -0.122 | 0.659 | 3 | 10 |
| Shift Task | 0.476 | -0.056 | 0.771 | 13 | 401 |
| Angling Risk Task Always Sunny | 0.428 | 0.039 | 0.636 | 6 | 21 |
| Motor Selective Stop Signal | 0.423 | -0.357 | 0.798 | 4 | 63 |
| Probabilistic Selection | 0.402 | -0.194 | 0.752 | 7 | 54 |
| Writing Task | 0.4 | 0.207 | 0.538 | 2 | 1 |
| Bickel Titrator | 0.066 | -0.033 | 0.783 | 4 | 30 |
Number of trials
Does number of items in a task have a significant effect on the average ICC of (mostly) raw measures for all trials from a task? No. (no effect on Spearman either)
tmp = measure_labels %>%
mutate(dv = as.character(dv)) %>%
filter(task == 'task') %>%
# left_join(rel_df[,c("dv", "spearman","icc")], by='dv') %>%
left_join(boot_df[,c("dv", "spearman","icc")], by='dv') %>%
filter(overall_difference != 'difference' & raw_fit %in% c('EZ', 'hddm') == FALSE)%>%
separate(dv, c('task_name', 'extra_1', 'extra_2'), sep = '\\.',remove=FALSE) %>%
select(-extra_1, -extra_2)
# summary(lm(icc ~ num_all_trials, data = tmp))
summary(lmerTest::lmer(icc ~ num_all_trials + (1|dv), data = tmp))Linear mixed model fit by REML t-tests use Satterthwaite approximations
to degrees of freedom [lmerMod]
Formula: icc ~ num_all_trials + (1 | dv)
Data: tmp
REML criterion at convergence: -408213
Scaled residuals:
Min 1Q Median 3Q Max
-12.548 -0.460 0.037 0.516 7.553
Random effects:
Groups Name Variance Std.Dev.
dv (Intercept) 0.04556 0.2135
Residual 0.00555 0.0745
Number of obs: 174000, groups: dv, 174
Fixed effects:
Estimate Std. Error df t value Pr(>|t|)
(Intercept) 5.81e-01 2.11e-02 1.72e+02 27.60 <2e-16 ***
num_all_trials 2.12e-05 1.19e-04 1.72e+02 0.18 0.86
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
Correlation of Fixed Effects:
(Intr)
num_ll_trls -0.640measure_labels %>%
mutate(dv = as.character(dv)) %>%
filter(task == 'task') %>%
# left_join(rel_df[,c("dv", "spearman","icc")], by='dv') %>%
left_join(boot_df[,c("dv", "spearman","icc")], by='dv') %>%
filter(overall_difference != 'difference' & raw_fit %in% c('EZ', 'hddm') == FALSE)%>%
separate(dv, c('task_name', 'extra_1', 'extra_2'), sep = '\\.',remove=FALSE) %>%
select(-extra_1, -extra_2) %>%
ggplot(aes(num_all_trials, icc))+
geom_point()+
geom_smooth(method="lm")+
theme_bw()+
xlab("Number of trials")+
ylab("ICC")
Trial number dependence intrameasure
The above analysis was looking at the effect of number of trials across tasks. But some tasks might be bad for individual difference measurement regardless of how many trials there are in them whereas for others fewer trials might be yielding a sufficiently reliable measure.
For tasks for which dependent variables are estimated using many trials one can ask: Does the same measure get less reliable if fewer trials are used to estimate its reliability?
This won’t make sense for all tasks. For example to estimate a risk aversion parameter you need all trials for Holt and Laury. For Kirby and Bickel you have specific conditions looking at fewer trials. The Cognitive Reflection Task might be more appropriate to analyze each item seaprately. The writing task does not have trial numbers. For all others it might be interesting to investigate.
These kinds of analyses are too task-specific and in-depth for a paper that is trying to give a global sense of the differences between self-regulation measures in their suitablity for individual difference analyses based on their stability across time. Such analyses would provide a detailed examination of how to extract the most reliable/best individual difference measure from tasks with a set of mediocre variables to begin with. Though we do not provide such a comprehensive analysis of this sort in this paper we provide a single example of this approach and hope the open access we provide to the data spurs further work.
For this example we look at the retest reliability of the three dependent measures (average accuracy, median response time, total score) from the hierarchical rule task with 360 trials. Here is a graph of how the point estimates of the retest reliability changes for each of the dependent measures using different numbers of trials to estimate them. While the reliability estimate for each of the variables respectively are …, … and … using all of the trials, these values are independent of number of trials used in the task for only certain variables. Based on this analysis researchers might decide to use a version of the task with fewer trials or only measures with consistently high reliability estimates.
Example task: three by two
Post process dv’s from cluster to calculate reliabilities
t1_dvs = read.csv(paste0(retest_data_path, 't1_tbt_dvs.csv'))
t2_dvs = read.csv(paste0(retest_data_path, 't2_tbt_dvs.csv'))
t1_dvs = t1_dvs %>% select(-X)
t2_dvs = t2_dvs %>% select(-X)hr_merge = merge(t1_dvs, t2_dvs, by = c("sub_id", "breaks"))
hr_merge = hr_merge %>%
gather(key, value, -sub_id, -breaks) %>%
separate(key, c("dv", "time"), sep="\\.") %>%
mutate(time = ifelse(time == "x", 1, 2))
t1_dvs = hr_merge %>%
filter(time == 1) %>%
select(-time) %>%
spread(dv, value)
t2_dvs = hr_merge %>%
filter(time == 2) %>%
select(-time) %>%
spread(dv, value)
# calculate point estimates for reliability of each of the variables for each break
# get_icc for each break of tmp_t1_dvs and tmp_t2_dvs
trial_num_rel_df = data.frame(breaks=rep(NA, length(unique(t1_dvs$breaks))),
acc_icc=rep(NA, length(unique(t1_dvs$breaks))),
avg_rt_error_icc=rep(NA, length(unique(t1_dvs$breaks))),
std_rt_error_icc=rep(NA, length(unique(t1_dvs$breaks))),
avg_rt_icc=rep(NA, length(unique(t1_dvs$breaks))),
std_rt_icc=rep(NA, length(unique(t1_dvs$breaks))),
missed_percent_icc=rep(NA, length(unique(t1_dvs$breaks))),
cue_switch_cost_rt_100_icc=rep(NA, length(unique(t1_dvs$breaks))),
cue_switch_cost_rt_900_icc=rep(NA, length(unique(t1_dvs$breaks))),
task_switch_cost_rt_100_icc=rep(NA, length(unique(t1_dvs$breaks))),
task_switch_cost_rt_900_icc=rep(NA, length(unique(t1_dvs$breaks))),
cue_switch_cost_acc_100_icc=rep(NA, length(unique(t1_dvs$breaks))),
cue_switch_cost_acc_900_icc=rep(NA, length(unique(t1_dvs$breaks))),
task_switch_cost_acc_100_icc=rep(NA, length(unique(t1_dvs$breaks))),
task_switch_cost_acc_900_icc=rep(NA, length(unique(t1_dvs$breaks))))
for(i in 1:length(unique(t1_dvs$breaks))){
cur_break = unique(t1_dvs$breaks)[i]
tmp_t1_dvs = t1_dvs %>% filter(breaks == cur_break)
tmp_t2_dvs = t2_dvs %>% filter(breaks == cur_break)
trial_num_rel_df$breaks[i] = cur_break
trial_num_rel_df$acc_icc[i] = get_icc("acc", tmp_t1_dvs, tmp_t2_dvs)
trial_num_rel_df$avg_rt_error_icc[i] = get_icc("avg_rt_error", tmp_t1_dvs, tmp_t2_dvs)
trial_num_rel_df$std_rt_error_icc[i] = get_icc("std_rt_error", tmp_t1_dvs, tmp_t2_dvs)
trial_num_rel_df$avg_rt_icc[i] = get_icc("avg_rt", tmp_t1_dvs, tmp_t2_dvs)
trial_num_rel_df$std_rt_icc[i] = get_icc("std_rt", tmp_t1_dvs, tmp_t2_dvs)
trial_num_rel_df$missed_percent_icc[i] = get_icc("missed_percent", tmp_t1_dvs, tmp_t2_dvs)
trial_num_rel_df$cue_switch_cost_rt_100_icc[i] = get_icc("cue_switch_cost_rt_100", tmp_t1_dvs, tmp_t2_dvs)
trial_num_rel_df$cue_switch_cost_rt_900_icc[i] = get_icc("cue_switch_cost_rt_900", tmp_t1_dvs, tmp_t2_dvs)
trial_num_rel_df$trial_switch_cost_rt_100_icc[i] = get_icc("task_switch_cost_rt_100", tmp_t1_dvs, tmp_t2_dvs)
trial_num_rel_df$trial_switch_cost_rt_900_icc[i] = get_icc("task_switch_cost_rt_900", tmp_t1_dvs, tmp_t2_dvs)
trial_num_rel_df$cue_switch_cost_acc_100_icc[i] = get_icc("cue_switch_cost_acc_100", tmp_t1_dvs, tmp_t2_dvs)
trial_num_rel_df$cue_switch_cost_acc_900_icc[i] = get_icc("cue_switch_cost_acc_900", tmp_t1_dvs, tmp_t2_dvs)
trial_num_rel_df$trial_switch_cost_acc_100_icc[i] = get_icc("task_switch_cost_acc_100", tmp_t1_dvs, tmp_t2_dvs)
trial_num_rel_df$trial_switch_cost_acc_900_icc[i] = get_icc("task_switch_cost_acc_900", tmp_t1_dvs, tmp_t2_dvs)
}
rm(i, cur_break, tmp_t1_dvs, tmp_t2_dvs)
trial_num_rel_df$breaks = as.numeric(trial_num_rel_df$breaks)
write.csv(trial_num_rel_df, paste0(retest_data_path, 'trial_num_rel_df_tbt.csv'))# trial_num_rel_df = read.csv(paste0(retest_data_path, 'trial_num_rel_df_tbt.csv'))
cols <- c("Accuracy" = '#084594', "Average RT error" = '#99000d', "SD RT error" = '#cb181d', "Average RT correct" = '#ef3b2c', "SD RT correct" = '#fb6a4a', "Missed percentage" = '#2171b5', "Cue switch cost RT (100)" = '#fc9272', "Cue switch cost RT (900)" = '#fcbba1', "Task switch cost RT (100)" = '#fee0d2', "Task switch cost RT (900)" = '#fff5f0', "Cue switch cost Acc (100)" = '#4292c6', "Cue switch cost Acc (900)" = '#6baed6', "Task switch cost Acc (100)" = '#9ecae1', "Task switch cost Acc (900)" = '#c6dbef')
trial_num_rel_df %>%
gather(key, value, -breaks) %>%
ggplot(aes(breaks*10, value, color=factor(key, levels = c("acc_icc", "avg_rt_error_icc", "std_rt_error_icc", "avg_rt_icc", "std_rt_icc", "missed_percent_icc", "cue_switch_cost_rt_100_icc", "cue_switch_cost_rt_900_icc", "task_switch_cost_rt_100_icc", "task_switch_cost_rt_900_icc", "cue_switch_cost_acc_100_icc", "cue_switch_cost_acc_900_icc", "task_switch_cost_acc_100_icc", "task_switch_cost_acc_900_icc"), labels = c("Accuracy", "Average RT error", "SD RT error", "Average RT correct", "SD RT correct", "Missed percentage", "Cue switch cost RT (100)", "Cue switch cost RT (900)", "Task switch cost RT (100)", "Task switch cost RT (900)", "Cue switch cost Acc (100)", "Cue switch cost Acc (900)", "Task switch cost Acc (100)", "Task switch cost Acc (900)"))))+
geom_point()+
geom_line()+
theme_bw()+
xlab("Number of trials")+
ylab("ICC")+
theme(legend.title = element_blank(),
legend.position = 'bottom')+
scale_color_manual(values = cols,
breaks = c("Average RT error", "Accuracy", "SD RT error", "Missed percentage", "Average RT correct", "Cue switch cost Acc (100)", "SD RT correct", "Cue switch cost Acc (900)", "Cue switch cost RT (100)", "Task switch cost Acc (100)", "Task switch cost RT (100)", "Task switch cost Acc (900)", "Task switch cost RT (900)"))+
ylim(-1,1)+
guides(color = guide_legend(ncol=2, byrow=T))
ggsave('Intrameasure_Trialnum_Dependendence.jpg', device = "jpeg", path = "/Users/zeynepenkavi/Dropbox/PoldrackLab/SRO_Retest_Analyses/output/figures", width = 10, height = 5, units = "in", dpi = 100)Raw vs DDM
Checking DDM results in the bootstrapped estimates. Variables using all trials are significantly more reliable compared to difference scores. Raw measures don’t differ from DDM parameters. Which DDM is better depends on whether all trials are used.
tmp = measure_labels %>%
mutate(dv = as.character(dv)) %>%
filter(ddm_task == 1,
overall_difference != "condition",
rt_acc != 'other') %>%
drop_na() %>%
left_join(boot_df[,c("dv", "icc", "spearman")], by = 'dv')
tmp %>%
drop_na() %>% #try removing this in final release
group_by(overall_difference, raw_fit, rt_acc) %>%
summarise(icc_median = quantile(icc, probs = 0.5),
icc_2.5 = quantile(icc, probs = 0.025),
icc_97.5 = quantile(icc, probs = 0.975),
spearman_median = quantile(spearman, probs = 0.5),
spearman_2.5 = quantile(spearman, probs = 0.025),
spearman_97.5 = quantile(spearman, probs = 0.975),
num_vars = n()/1000) %>%
datatable() %>%
formatRound(columns=c('icc_median', 'icc_2.5', 'icc_97.5', 'spearman_median', 'spearman_2.5', 'spearman_97.5'), digits=3)tmp_save = tmp %>%
drop_na() %>% #try removing this in final release
group_by(overall_difference, raw_fit, rt_acc) %>%
summarise(icc_median = quantile(icc, probs = 0.5),
icc_2.5 = quantile(icc, probs = 0.025),
icc_97.5 = quantile(icc, probs = 0.975),
num_vars = n()/1000) %>%
ungroup() %>%
mutate(overall_difference = as.character(overall_difference),
raw_fit = as.character(raw_fit),
rt_acc = as.character(rt_acc)) %>%
arrange(-icc_median)
sjt.df(tmp_save %>% mutate_if(is.numeric, funs(round(., 3))), describe=F, hide.progress = TRUE, show.rownames = FALSE, file = "/Users/zeynepenkavi/Dropbox/PoldrackLab/SRO_Retest_Analyses/output/tables/ddm_rel_table.doc")| overall_difference | raw_fit | rt_acc | icc_median | icc_2.5 | icc_97.5 | num_vars |
|---|---|---|---|---|---|---|
| overall | raw | rt | 0.736 | 0.535 | 0.854 | 16 |
| overall | EZ | drift rate | 0.724 | 0.528 | 0.826 | 14 |
| overall | raw | accuracy | 0.684 | -0.375 | 0.832 | 14 |
| overall | hddm | drift rate | 0.684 | 0.387 | 0.816 | 16 |
| overall | EZ | non-decision | 0.647 | 0.197 | 0.794 | 14 |
| overall | EZ | threshold | 0.624 | 0.384 | 0.823 | 14 |
| overall | hddm | threshold | 0.544 | 0.316 | 0.668 | 14 |
| overall | hddm | non-decision | 0.449 | -0.032 | 0.671 | 14 |
| difference | hddm | drift rate | 0.404 | -0.242 | 0.651 | 18 |
| difference | raw | rt | 0.299 | -0.193 | 0.63 | 25 |
| difference | raw | accuracy | 0.27 | -0.245 | 0.698 | 13 |
| difference | EZ | drift rate | 0.248 | -0.196 | 0.579 | 17 |
| difference | EZ | non-decision | 0.236 | -0.204 | 0.652 | 17 |
| difference | EZ | threshold | 0.097 | -0.449 | 0.537 | 17 |
| difference | hddm | threshold | 0.09 | -0.201 | 0.345 | 1 |
Comparing overall vs difference: overall has higher reliability than difference.
summary(lmerTest::lmer(icc ~ overall_difference + (1|dv) ,tmp))Linear mixed model fit by REML t-tests use Satterthwaite approximations
to degrees of freedom [lmerMod]
Formula: icc ~ overall_difference + (1 | dv)
Data: tmp
REML criterion at convergence: -401676
Scaled residuals:
Min 1Q Median 3Q Max
-11.245 -0.467 0.041 0.521 6.005
Random effects:
Groups Name Variance Std.Dev.
dv (Intercept) 0.03486 0.1867
Residual 0.00966 0.0983
Number of obs: 224000, groups: dv, 224
Fixed effects:
Estimate Std. Error df t value Pr(>|t|)
(Intercept) 0.245 0.018 221.400 13.6 <2e-16 ***
overall_differenceoverall 0.368 0.025 221.400 14.8 <2e-16 ***
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
Correlation of Fixed Effects:
(Intr)
ovrll_dffrn -0.720“consistent with the summing of variance in the difference score”
tmp = measure_labels %>%
mutate(dv = as.character(dv)) %>%
filter(ddm_task == 1,
overall_difference != "condition",
rt_acc != 'other') %>%
drop_na() %>%
left_join(boot_df[,c("dv", "icc", "var_subs","var_ind", "var_resid")], by = 'dv') %>%
mutate(var_subs_pct = var_subs/(var_subs+var_ind+var_resid),
var_ind_pct = var_ind/(var_subs+var_ind+var_resid),
var_resid_pct = var_resid/(var_subs+var_ind+var_resid)) %>%
select(-task, -ddm_task, -num_all_trials, -var_resid, -var_subs, -var_ind) %>%
gather(key, value, -dv, -overall_difference, -raw_fit, -rt_acc, -icc)
tmp %>%
filter(key == "var_resid_pct") %>%
ggplot(aes(raw_fit, value, fill=overall_difference))+
geom_boxplot()
summary(lmer(value ~ overall_difference*raw_fit + (1|dv), data=tmp %>% filter(key=="var_resid_pct")))Linear mixed model fit by REML ['lmerMod']
Formula: value ~ overall_difference * raw_fit + (1 | dv)
Data: tmp %>% filter(key == "var_resid_pct")
REML criterion at convergence: -608085
Scaled residuals:
Min 1Q Median 3Q Max
-5.529 -0.543 0.020 0.601 5.547
Random effects:
Groups Name Variance Std.Dev.
dv (Intercept) 0.00539 0.0734
Residual 0.00385 0.0620
Number of obs: 224000, groups: dv, 224
Fixed effects:
Estimate Std. Error t value
(Intercept) 0.3437 0.0103 33.4
overall_differenceoverall -0.1594 0.0153 -10.4
raw_fithddm -0.1070 0.0197 -5.4
raw_fitraw -0.0320 0.0157 -2.0
overall_differenceoverall:raw_fithddm 0.1270 0.0253 5.0
overall_differenceoverall:raw_fitraw 0.0240 0.0236 1.0
Correlation of Fixed Effects:
(Intr) ovrll_ rw_fth rw_ftr
ovrll_dffrn -0.672
raw_fithddm -0.521 0.350
raw_fitraw -0.653 0.439 0.340
ovrll_dffrncvrll:rw_fth 0.406 -0.605 -0.780 -0.265
ovrll_dffrncvrll:rw_ftr 0.436 -0.649 -0.227 -0.668
ovrll_dffrncvrll:rw_fth
ovrll_dffrn
raw_fithddm
raw_fitraw
ovrll_dffrncvrll:rw_fth
ovrll_dffrncvrll:rw_ftr 0.392 #Simple effects
summary(lmer(value ~ overall_difference + (1|dv), data=tmp %>% filter(key=="var_resid_pct", raw_fit== "EZ")))Linear mixed model fit by REML ['lmerMod']
Formula: value ~ overall_difference + (1 | dv)
Data: tmp %>% filter(key == "var_resid_pct", raw_fit == "EZ")
REML criterion at convergence: -241956
Scaled residuals:
Min 1Q Median 3Q Max
-5.224 -0.536 0.041 0.607 4.544
Random effects:
Groups Name Variance Std.Dev.
dv (Intercept) 0.00448 0.0669
Residual 0.00431 0.0657
Number of obs: 93000, groups: dv, 93
Fixed effects:
Estimate Std. Error t value
(Intercept) 0.34371 0.00938 36.7
overall_differenceoverall -0.15937 0.01395 -11.4
Correlation of Fixed Effects:
(Intr)
ovrll_dffrn -0.672summary(lmer(value ~ overall_difference + (1|dv), data=tmp %>% filter(key=="var_resid_pct", raw_fit== "raw")))Linear mixed model fit by REML ['lmerMod']
Formula: value ~ overall_difference + (1 | dv)
Data: tmp %>% filter(key == "var_resid_pct", raw_fit == "raw")
REML criterion at convergence: -184635
Scaled residuals:
Min 1Q Median 3Q Max
-4.952 -0.522 0.021 0.590 4.769
Random effects:
Groups Name Variance Std.Dev.
dv (Intercept) 0.00525 0.0724
Residual 0.00385 0.0620
Number of obs: 68000, groups: dv, 68
Fixed effects:
Estimate Std. Error t value
(Intercept) 0.3117 0.0118 26.51
overall_differenceoverall -0.1354 0.0177 -7.65
Correlation of Fixed Effects:
(Intr)
ovrll_dffrn -0.664summary(lmer(value ~ overall_difference + (1|dv), data=tmp %>% filter(key=="var_resid_pct", raw_fit== "hddm")))Linear mixed model fit by REML ['lmerMod']
Formula: value ~ overall_difference + (1 | dv)
Data: tmp %>% filter(key == "var_resid_pct", raw_fit == "hddm")
REML criterion at convergence: -183232
Scaled residuals:
Min 1Q Median 3Q Max
-5.198 -0.592 -0.008 0.605 6.113
Random effects:
Groups Name Variance Std.Dev.
dv (Intercept) 0.00689 0.0830
Residual 0.00317 0.0563
Number of obs: 63000, groups: dv, 63
Fixed effects:
Estimate Std. Error t value
(Intercept) 0.2368 0.0190 12.43
overall_differenceoverall -0.0324 0.0228 -1.42
Correlation of Fixed Effects:
(Intr)
ovrll_dffrn -0.836Checking Rogosa’s claim that ‘Difference scores are reliable when individual differences in true change exist.’ How do we measure ‘individual difference in true change’ 1.between subject variability in difference score distributions 2.within subject variance in icc Using 2 (because comparable between subject variance across different measures would be hard) Expecting to see: positive correlation between mean icc and mean var_ind_pct Result: The correlation between mean icc and mean var ind pct is NOT significant BUT looking at the distribution of mean var ind pct there is not a lot of variability t
tmp = tmp %>%
spread(key, value) %>%
group_by(dv) %>%
summarise(mean_var_ind_pct = mean(var_ind_pct),
mean_var_subs_pct = mean(var_subs_pct),
mean_icc = mean(icc)) %>%
left_join(measure_labels, by='dv')Warning: Column `dv` joining character vector and factor, coercing into
character vectortmp %>%
filter(overall_difference == 'difference') %>%
ggplot(aes(mean_var_ind_pct, mean_icc))+
geom_point()+
geom_abline(slope=1, intercept=0)+
geom_smooth(method = "lm")
summary(lm(mean_icc ~ mean_var_ind_pct,data=tmp %>%
filter(overall_difference == 'difference')))
Call:
lm(formula = mean_icc ~ mean_var_ind_pct, data = tmp %>% filter(overall_difference ==
"difference"))
Residuals:
Min 1Q Median 3Q Max
-0.6262 -0.1294 -0.0076 0.1613 0.3764
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) 0.2302 0.0366 6.29 7.1e-09 ***
mean_var_ind_pct 0.0587 0.1197 0.49 0.62
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
Residual standard error: 0.212 on 106 degrees of freedom
Multiple R-squared: 0.00227, Adjusted R-squared: -0.00715
F-statistic: 0.241 on 1 and 106 DF, p-value: 0.625tmp %>%
filter(overall_difference == 'difference') %>%
ggplot(aes(mean_var_ind_pct))+
geom_density()
summary(tmp$mean_var_ind_pct[tmp$overall_difference == "difference"]) Min. 1st Qu. Median Mean 3rd Qu. Max.
0.0917 0.1350 0.1820 0.2540 0.3350 0.9560 summary(tmp$mean_var_subs_pct[tmp$overall_difference == "difference"]) Min. 1st Qu. Median Mean 3rd Qu. Max.
0.0255 0.3690 0.4440 0.4330 0.5170 0.6300 tmp = tmp %>% filter(overall_difference == 'difference')
t.test(tmp$mean_var_ind_pct, tmp$mean_var_subs_pct, paired=T)
Paired t-test
data: tmp$mean_var_ind_pct and tmp$mean_var_subs_pct
t = -6.7, df = 110, p-value = 8e-10
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
-0.2318 -0.1265
sample estimates:
mean of the differences
-0.1791 tmp = measure_labels %>%
mutate(dv = as.character(dv)) %>%
filter(ddm_task == 1,
overall_difference != "condition",
rt_acc != 'other') %>%
drop_na() %>%
left_join(boot_df[,c("dv", "icc", "spearman")], by = 'dv')Comparing raw vs ddm in overall estimates: EZ is significantly better than HDDM and comparable to raw estimates.
summary(lmerTest::lmer(icc ~ raw_fit + (1|dv) ,tmp %>% filter(overall_difference == "overall")))Linear mixed model fit by REML t-tests use Satterthwaite approximations
to degrees of freedom [lmerMod]
Formula: icc ~ raw_fit + (1 | dv)
Data: tmp %>% filter(overall_difference == "overall")
REML criterion at convergence: -278565
Scaled residuals:
Min 1Q Median 3Q Max
-11.366 -0.485 0.031 0.510 8.141
Random effects:
Groups Name Variance Std.Dev.
dv (Intercept) 0.02299 0.1516
Residual 0.00526 0.0725
Number of obs: 116000, groups: dv, 116
Fixed effects:
Estimate Std. Error df t value Pr(>|t|)
(Intercept) 0.6493 0.0234 113.4000 27.75 <2e-16 ***
raw_fithddm -0.1068 0.0327 113.4000 -3.27 0.0014 **
raw_fitraw 0.0186 0.0362 113.4000 0.51 0.6080
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
Correlation of Fixed Effects:
(Intr) rw_fth
raw_fithddm -0.715
raw_fitraw -0.645 0.462Comparing raw vs ddm in difference scores: EZ is significantly worse than HDDM and comparable to raw estimates.
summary(lmerTest::lmer(icc ~ raw_fit + (1|dv) ,tmp %>% filter(overall_difference == "difference")))Linear mixed model fit by REML t-tests use Satterthwaite approximations
to degrees of freedom [lmerMod]
Formula: icc ~ raw_fit + (1 | dv)
Data: tmp %>% filter(overall_difference == "difference")
REML criterion at convergence: -150642
Scaled residuals:
Min 1Q Median 3Q Max
-9.214 -0.535 0.065 0.615 4.172
Random effects:
Groups Name Variance Std.Dev.
dv (Intercept) 0.0423 0.206
Residual 0.0144 0.120
Number of obs: 108000, groups: dv, 108
Fixed effects:
Estimate Std. Error df t value Pr(>|t|)
(Intercept) 0.1900 0.0288 105.1000 6.59 1.8e-09 ***
raw_fithddm 0.1425 0.0553 105.1000 2.58 0.011 *
raw_fitraw 0.0855 0.0441 105.1000 1.94 0.055 .
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
Correlation of Fixed Effects:
(Intr) rw_fth
raw_fithddm -0.521
raw_fitraw -0.653 0.340tmp %>%
ggplot(aes(factor(raw_fit, levels = c("raw", "EZ", "hddm"), labels=c("Raw", "EZ-diffusion", "Hierarchical diffusion")), icc, fill=factor(rt_acc, levels = c("rt","accuracy", "drift rate", "threshold", "non-decision"), labels=c("Response Time", "Accuracy","Drift Rate", "Threshold", "Non-decision"))))+
geom_boxplot()+
facet_wrap(~factor(overall_difference, levels=c("overall", "difference"), labels=c("Overall", "Difference")))+
theme_bw()+
ylab("ICC")+
xlab("")+
theme(legend.title = element_blank(),
legend.position = 'bottom',
legend.text = element_text(size=16),
strip.text = element_text(size=16),
axis.text = element_text(size = 16),
text = element_text(size=16))+
guides(fill = guide_legend(ncol = 2))+
scale_fill_discrete(breaks=c("Drift Rate", "Threshold", "Non-decision", "Response Time", "Accuracy"))
ggsave('Bootstrap_DDM_Comp.jpg', device = "jpeg", path = "/Users/zeynepenkavi/Dropbox/PoldrackLab/SRO_Retest_Analyses/output/figures/", width = 14, height = 10, units = "in", limitsize = FALSE)tmp2 = measure_labels %>%
mutate(dv = as.character(dv)) %>%
filter(ddm_task == 1,
overall_difference != "condition",
rt_acc != "other") %>%
drop_na() %>%
left_join(rel_df[,c("dv", "icc", "spearman")], by = 'dv')
tmp2 %>%
ggplot(aes(factor(raw_fit, levels = c("raw", "EZ", "hddm"), labels=c("Raw", "EZ-diffusion", "Hierarchical diffusion")), icc, fill=factor(rt_acc, levels = c("rt","accuracy", "drift rate", "threshold", "non-decision"), labels=c("Response Time", "Accuracy","Drift Rate", "Threshold", "Non-decision"))))+
geom_boxplot()+
facet_wrap(~factor(overall_difference, levels=c("overall", "difference"), labels=c("Overall", "Difference")))+
theme_bw()+
ylab("ICC")+
xlab("")+
theme(legend.title = element_blank(),
legend.position = 'bottom',
legend.text = element_text(size=16),
strip.text = element_text(size=16),
axis.text = element_text(size = 16),
text = element_text(size=16))+
guides(fill = guide_legend(ncol = 2))+
scale_fill_discrete(breaks=c("Drift Rate", "Threshold", "Non-decision", "Response Time", "Accuracy"))
ggsave('PointEst_DDM_Comp.jpg', device = "jpeg", path = "/Users/zeynepenkavi/Dropbox/PoldrackLab/SRO_Retest_Analyses/output/figures/", width = 14, height = 10, units = "in", limitsize = FALSE)Survey reliabilities
tmp = measure_labels %>%
mutate(dv = as.character(dv)) %>%
filter(task == 'survey') %>%
left_join(boot_df[,c("dv", "icc", "spearman")], by = 'dv') %>%
filter(overall_difference != 'difference' & raw_fit %in% c('EZ', 'hddm') == FALSE)%>%
separate(dv, c('task_name', 'extra_1', 'extra_2'), sep = '\\.',remove=FALSE) %>%
select(-extra_1, -extra_2) %>%
group_by(task_name) %>%
summarise(median_icc = median(icc),
median_spearman = median(spearman),
icc_2.5 = quantile(icc, probs = 0.025),
icc_97.5 = quantile(icc, probs = 0.975),
min_spearman = min(spearman),
max_spearman = max(spearman),
num_measures = n()/1000,
mean_num_trials = round(mean(num_all_trials)))%>%
arrange(-median_icc, -median_spearman)
tmp %>%
datatable() %>%
formatRound(columns=c('median_spearman', 'median_icc',
'min_spearman', 'icc_2.5',
'max_spearman', 'icc_97.5'), digits=3)tmp = tmp%>%
select(-min_spearman, -max_spearman,-median_spearman) %>%
mutate(task_name = gsub("_", " ", task_name),
task_name = gsub("(^|[[:space:]])([[:alpha:]])", "\\1\\U\\2", task_name, perl=TRUE),
task_name = gsub("Survey", "", task_name))
names(tmp) = gsub("_", " ", names(tmp))
names(tmp) = gsub("(^|[[:space:]])([[:alpha:]])", "\\1\\U\\2", names(tmp), perl=TRUE)
sjt.df(tmp %>% mutate_if(is.numeric, funs(round(., 3))), describe=F, hide.progress = TRUE, show.rownames = FALSE, file = "/Users/zeynepenkavi/Dropbox/PoldrackLab/SRO_Retest_Analyses/output/tables/survey_rel_table.doc")| Task Name | Median Icc | Icc 2.5 | Icc 97.5 | Num Measures | Mean Num Trials |
|---|---|---|---|---|---|
| Self Regulation | 0.95 | 0.936 | 0.961 | 1 | 31 |
| Grit Scale | 0.942 | 0.924 | 0.958 | 1 | 8 |
| Brief Self Control | 0.941 | 0.92 | 0.966 | 1 | 13 |
| Ten Item Personality | 0.936 | 0.852 | 0.966 | 5 | 2 |
| Time Perspective | 0.92 | 0.865 | 0.954 | 5 | 11 |
| Five Facet Mindfulness | 0.909 | 0.828 | 0.965 | 6 | 13 |
| Bis Bas | 0.904 | 0.851 | 0.958 | 5 | 7 |
| Dospert Rt | 0.903 | 0.855 | 0.953 | 5 | 6 |
| Impulsive Venture | 0.9 | 0.815 | 0.95 | 2 | 17 |
| Sensation Seeking | 0.899 | 0.852 | 0.951 | 5 | 16 |
| Mindful Attention Awareness | 0.897 | 0.862 | 0.928 | 1 | 15 |
| Erq | 0.896 | 0.827 | 0.933 | 2 | 5 |
| Upps Impulsivity | 0.886 | 0.718 | 0.959 | 5 | 12 |
| Mpq Control | 0.879 | 0.796 | 0.95 | 1 | 24 |
| Eating | 0.872 | 0.813 | 0.914 | 4 | 9 |
| Bis11 | 0.864 | 0.63 | 0.939 | 4 | 15 |
| Dickman | 0.85 | 0.721 | 0.91 | 2 | 12 |
| Future Time Perspective | 0.844 | 0.792 | 0.903 | 1 | 10 |
| Dospert Rp | 0.842 | 0.761 | 0.901 | 5 | 6 |
| Dospert Eb | 0.8 | 0.711 | 0.868 | 5 | 6 |
| Selection Optimization Compensation | 0.799 | 0.598 | 0.898 | 4 | 12 |
| Leisure Time Activity | 0.798 | 0.731 | 0.857 | 1 | 1 |
| Theories Of Willpower | 0.794 | 0.725 | 0.86 | 1 | 12 |
Number of items
Does number of items in a survey have a significant effect on the average ICC of survey measures? No.
tmp = measure_labels %>%
mutate(dv = as.character(dv)) %>%
filter(task == 'survey') %>%
# left_join(rel_df[,c("dv", "spearman","icc")], by='dv') %>%
left_join(boot_df[,c("dv", "spearman","icc")], by='dv') %>%
filter(overall_difference != 'difference' & raw_fit %in% c('EZ', 'hddm') == FALSE)%>%
separate(dv, c('task_name', 'extra_1', 'extra_2'), sep = '\\.',remove=FALSE) %>%
select(-extra_1, -extra_2)
# summary(lm(icc ~ num_all_trials, data = tmp))
summary(lmerTest::lmer(icc ~ num_all_trials + (1|dv), data = tmp))Linear mixed model fit by REML t-tests use Satterthwaite approximations
to degrees of freedom [lmerMod]
Formula: icc ~ num_all_trials + (1 | dv)
Data: tmp
REML criterion at convergence: -320933
Scaled residuals:
Min 1Q Median 3Q Max
-10.265 -0.516 -0.004 0.540 6.540
Random effects:
Groups Name Variance Std.Dev.
dv (Intercept) 0.003327 0.0577
Residual 0.000673 0.0259
Number of obs: 72000, groups: dv, 72
Fixed effects:
Estimate Std. Error df t value Pr(>|t|)
(Intercept) 0.864206 0.011510 70.400000 75.08 <2e-16 ***
num_all_trials 0.001071 0.000915 70.400000 1.17 0.25
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
Correlation of Fixed Effects:
(Intr)
num_ll_trls -0.807measure_labels %>%
mutate(dv = as.character(dv)) %>%
filter(task == 'task') %>%
# left_join(rel_df[,c("dv", "spearman","icc")], by='dv') %>%
left_join(boot_df[,c("dv", "spearman","icc")], by='dv') %>%
filter(overall_difference != 'difference' & raw_fit %in% c('EZ', 'hddm') == FALSE)%>%
separate(dv, c('task_name', 'extra_1', 'extra_2'), sep = '\\.',remove=FALSE) %>%
select(-extra_1, -extra_2) %>%
ggplot(aes(num_all_trials, icc))+
geom_point()+
geom_smooth(method="lm")+
theme_bw()+
xlab("Number of trials")+
ylab("ICC")
LS0tCnRpdGxlOiAnU2VsZiBSZWd1bGF0aW9uIE9udG9sb2d5IFJldGVzdCBBbmFseXNlcycKb3V0cHV0OgpnaXRodWJfZG9jdW1lbnQ6CnRvYzogeWVzCnRvY19mbG9hdDogeWVzCi0tLQoKYGBge3IsIG1lc3NhZ2U9RkFMU0UsIHdhcm5pbmc9RkFMU0UsIGluY2x1ZGU9RkFMU0V9CnNvdXJjZSgnL1VzZXJzL3pleW5lcGVua2F2aS9Ecm9wYm94L1BvbGRyYWNrTGFiL1NST19SZXRlc3RfQW5hbHlzZXMvY29kZS9TUk9fUmV0ZXN0X0FuYWx5c2VzX0hlbHBlcl9GdW5jdGlvbnMuUicpCnRoZW1lX3NldCh0aGVtZV9idygpKQoKdGVzdF9kYXRhX3BhdGggPSAnL1VzZXJzL3pleW5lcGVua2F2aS9Eb2N1bWVudHMvUG9sZHJhY2tMYWJMb2NhbC9TZWxmX1JlZ3VsYXRpb25fT250b2xvZ3kvRGF0YS9Db21wbGV0ZV8wMy0yOS0yMDE4LycKCnJldGVzdF9kYXRhX3BhdGggPSAnL1VzZXJzL3pleW5lcGVua2F2aS9Eb2N1bWVudHMvUG9sZHJhY2tMYWJMb2NhbC9TZWxmX1JlZ3VsYXRpb25fT250b2xvZ3kvRGF0YS9SZXRlc3RfMDMtMjktMjAxOC8nCmBgYAoKIyBJbnRyb2R1Y3Rpb24KClRoZSBwc3ljaG9sb2dpY2FsIGNvbnN0cnVjdCBvZiBzZWxmLXJlZ3VsYXRpb24sIG9yIHJlbGF0ZWQgY29uY2VwdHMgb2YgaW1wdWxzaXZpdHksIHNlbGYtY29udHJvbCwgaW5oaWJpdGlvbiBhbmQgb3RoZXJzIGNvcnJlbGF0ZSB3aXRoIHByb2JsZW1hdGljIHJlYWwtd29ybGQgYmVoYXZpb3JzIHN1Y2ggYXMgZGlzb3JkZXJlZCBlYXRpbmcgKE1vYmJzIGV0IGFsLiwgMjAxMCwgVmVyYmVrZW4gZXQgYWwuLCAyMDA5LCBOZWRlcmtvb3JuIGV0IGFsLiAyMDA2YSwgTmVkZXJrb29ybiBldCBhbC4sIDIwMDZiKSwgZ2FtYmxpbmcgKExhd3JlbmNlIGV0IGFsLiwgMjAwOSwgRnVlbnRlcyBldCBhbC4sIDIwMDYsIEFsZXNzaSBhbmQgUGV0cnksIDIwMDMpLCBkcnVnIGFkZGljdGlvbiAoQ29mZmV5IGV0IGFsLiwgMjAwMywgZGUgV2l0LCBDcmVhbiBhbmQgSm9obiwgMjAwMCwgU2hlciwgQmFydGhvbG93IGFuZCBXb29kLCAyMDAwLCBLaXJieSwgUGV0cnkgYW5kIEJpY2tlbCwgMTk5OSkgb3IgYmFkIGZpbmFuY2lhbCBkZWNpc2lvbnMgKE1laWVyIGFuZCBTcHJlbmdlciwgMjAxNSwgTWVpZXIgYW5kIFNwcmVuZ2VyLCAyMDEzLCBNZWllciBhbmQgU3ByZW5nZXIgMjAxMikuIEluIHRoZXNlIGxpbmVzIG9mIHdvcmsgbWVhc3VyZXMgb2Ygc2VsZi1yZWd1bGF0aW9uIGFyZSB1c2VkIGFzIGJlaGF2aW9yYWwgYXNzYXlzIGZvciBpbmRpdmlkdWFsIGRpZmZlcmVuY2UgYW5hbHlzZXMuIEFuIHVuZGVybHlpbmcgYXNzdW1wdGlvbiBvZiB0cmVhdGluZyBiZWhhdmlvcmFsIG1lYXN1cmVzIGFzIHJlZmxlY3RpdmUgb2YgcGVyc29uLXNwZWNpZmljIGNoYXJhY3RlcmlzdGljcyBpcyB0aGF0IHRoZSBtZWFzdXJlcyBvZiBzZWxmLXJlZ3VsYXRpb24gYXJlIHN0YWJsZSBhY3Jvc3MgdGltZS4gSW4gb3RoZXIgd29yZHMsIHRoYXQgdGhleSBoYXZlIGhpZ2ggdGVzdC1yZXRlc3QgcmVsaWFiaWxpdHkgKG9yIGhpZ2ggYmV0d2Vlbi0gYW5kIGxvdyB3aXRoaW4tc3ViamVjdCB2YXJpYWJpbGl0eSkuIFRoaXMgYXNzdW1wdGlvbiBpcyBub3QgZXh0ZW5zaXZlbHkgYW5kIGVxdWFsbHkgdGVzdGVkIGluIHRoZSBsaXRlcmF0dXJlIGZvciBhbGwgbWVhc3VyZXMgb2Ygc2VsZi1yZWd1bGF0aW9uLiAgICAKU2VsZi1yZWd1bGF0aW9uIG1lYXN1cmVzIGNhbiBiZSBncm91cGVkIGluIHR3byBicm9hZCBjYXRlZ29yaWVzOiBTdXJ2ZXlzIGFuZCBjb2duaXRpdmUgdGFza3MuIFdoaWxlIHRoZSBmb3JtZXIgdHlwaWNhbGx5IGdvZXMgdGhyb3VnaCBzb21lIGZvcm0gb2YgcHN5Y2hvbWV0cmljIHRlc3Rpbmcgb2Z0ZW4gZmlsdGVyaW5nIG91dCBpdGVtcyB3aXRoIGxvdyB0ZXN0LXJldGVzdCByZWxpYWJpbGl0eSB0aGlzIGlzIGxlc3MgZnJlcXVlbnRseSB0cnVlIGZvciB0aGUgbGF0dGVyLiBUaG91Z2ggc29tZSBlbXBpcmljYWwgcmVzdWx0cyBvbiB0ZXN0LXJldGVzdCByZWxpYWJpbGl0aWVzIG9mIGNlcnRhaW4gY29nbml0aXZlIHRhc2sgbWVhc3VyZXMgYXJlIHJlcG9ydGVkIGluIHBvY2tldHMgb2YgdGhlIGxpdGVyYXR1cmUgYW4gZXhoYXVzdGl2ZSBzdW1tYXJ5IG9yIHN5c3RlbWF0aWMgZWxpbWluYXRpb24gb2YgdGFza3Mgb3IgbWVhc3VyZXMgYmFzZWQgb24gc3RhYmlsaXR5IGFjcm9zcyB0aW1lIGRvZXMgbm90IHNlZW0gdG8gZXhpc3QuIFRlc3QtcmV0ZXN0IHJlbGlhYmlsaXR5IGlzLCBob3dldmVyLCBjcnVjaWFsIGZvciBpbmRpdmlkdWFsIGRpZmZlcmVuY2UgYW5hbHlzZXMgKEhlZGdlLCBQb3dlbGwgYW5kIFN1bW5lciwgMjAxNykuICAgClRvIGFuc3dlciB0aGUgcXVlc3Rpb24gb2YgcmVsaWFiaWxpdHkgb2Ygc2VsZi1yZWd1bGF0aW9uIG1lYXN1cmVzIGNvbXByZWhlbnNpdmVseSB3ZSBjcmVhdGVkIGEgbGFyZ2UgYmF0dGVyeSBjb25zaXN0aW5nIG9mIGJvdGggY29nbml0aXZlIHRhc2sgYW5kIHN1cnZleSBtZWFzdXJlcy4gSW4gdGhpcyBwYXBlciB3ZSBtYWtlIHR3byBjb250cmlidXRpb25zOiBGaXJzdCB3ZSBwcm92aWRlIGEgcmV2aWV3IG9mIHRoZXNlIG1lYXN1cmVzIGFsb25nIHdpdGggdGhlaXIgcmVwb3J0ZWQgcmVsaWFiaWxpdGllcyB3aGVuIGF2YWlsYWJsZS4gVGhlbiB3ZSByZXBvcnQgcmVzdWx0cyBvZiBhIGxhcmdlIHN0dWR5IHdlIGNvbmR1Y3RlZCB3aGVyZSB3ZSBjb2xsZWN0ZWQgcmV0ZXN0IHJlbGlhYmlsaXR5IGRhdGEgdXNpbmcgYWxsIG9mIHRoZXNlIG1lYXN1cmVzLiBUaGlzIGVmZm9ydCBub3Qgb25seSBmaWxscyBpbiBhIG5vdGFibGUgZ2FwIGluIHRoZSBsaXRlcmF0dXJlIGluIGNsYXJpZnlpbmcgd2hpY2ggbWVhc3VyZXMgb2Ygc2VsZi1yZWd1bGF0aW9uIGFyZSBzdGFibGUgYWNyb3NzIHRpbWUgYnV0IGFsc28gcHJvdmlkZXMgZ3VpZGFuY2Ugb24gZmFjdG9ycyB0byBwYXkgYXR0ZW50aW9uIHRvIHdoZW4gY29uc3RydWN0aW5nIG5ldyB0YXNrcyBmb3IgaW5kaXZpZHVhbCBkaWZmZXJlbmNlIG1lYXN1cmVzLiAgCgojIE1ldGhvZHMKCiMjIERhdGEgY29sbGVjdGlvbgoKVGhpcyBzYW1wbGUgY29uc2lzdHMgb2YgZGF0YSBmb3IgMTUwIHN1YmplY3RzIG9mIHRoZSBvcmlnaW5hbCBzYW1wbGUgb2YgNTIyIHRoYXQgaGFzIGNvbXBsZXRlZCB0aGUgaW5pdGlhbCBiYXR0ZXJ5IG9mIDM3IGNvZ25pdGl2ZSB0YXNrcywgMjMgc3VydmV5cyBhbmQgMyBzdXJ2ZXlzIG9uIGRlbW9ncmFwaGljcy4gRGV0YWlscyBvZiB0aGUgb3JpZ2luYWwgc2FtcGxlIGFzIHdlbGwgYXMgcXVhbGl0eSBjb250cm9sIChxYykgcHJvY2VkdXJlcyBhcmUgZGVzY3JpYmVkIGVsc2V3aGVyZSAoRWlzZW5iZXJnIGV0IGFsLiwgMjAxNykuIEludml0ZWQgcGFydGljaXBhbnRzIHdlcmUgY2hvc2VuIHJhbmRvbWx5IGFuZCBvbmx5IHN1YnNldHMgb2YgdGhlbSB3ZXJlIGludml0ZWQgZm9yIGEgZ2l2ZW4gYmF0Y2ggKGluc3RlYWQgb2Ygb3BlbmluZyB0aGUgYmF0dGVyeSB0byBhbGwgcXVhbGlmaWVkIHN1YmplY3RzKSB3aXRoIHRoZSBpbnRlbnRpb24gdG8gYXZvaWQgYSBwb3RlbnRpYWwgb3ZlcnNhbXBsaW5nIGFuZCBiaWFzIHRvd2FyZHMgImhpZ2ggc2VsZiByZWd1bGF0b3JzIi4KCmBgYHtyIHdhcm5pbmc9RkFMU0UsIG1lc3NhZ2U9RkFMU0V9CndvcmtlcnMgPSByZWFkLmNzdihwYXN0ZTAocmV0ZXN0X2RhdGFfcGF0aCwnTG9jYWwvVXNlcl83MTc1NzBfd29ya2Vycy5jc3YnKSkKd29ya2VycyA9IHdvcmtlcnMgJT4lIAogIGdyb3VwX2J5KFdvcmtlci5JRCkgJT4lCiAgbXV0YXRlKFJldGVzdF93b3JrZXI9aWZlbHNlKHN1bShDVVJSRU5ULlJldGVzdFdvcmtlcixDVVJSRU5ULlJldGVzdFdvcmtlckIyLENVUlJFTlQuUmV0ZXN0V29ya2VyQjMsQ1VSUkVOVC5SZXRlc3RXb3JrZXJCNCxDVVJSRU5ULlJldGVzdFdvcmtlckI1LG5hLnJtPVQpPjAsMSwwKSkgJT4lCiAgdW5ncm91cCgpCgp3b3JrZXJfY291bnRzIDwtIGZyb21KU09OKHBhc3RlMChyZXRlc3RfZGF0YV9wYXRoLCcvTG9jYWwvcmV0ZXN0X3dvcmtlcl9jb3VudHMuanNvbicpKQoKd29ya2VyX2NvdW50cyA9IGFzLmRhdGEuZnJhbWUodW5saXN0KHdvcmtlcl9jb3VudHMpKQpuYW1lcyh3b3JrZXJfY291bnRzKSA9ICJ0YXNrX2NvdW50IgpgYGAKCkluIHRvdGFsIGByIHN1bSh3b3JrZXJzJFJldGVzdF93b3JrZXIpYCBwYXJ0aWNpcGFudHMgd2VyZSBpbnZpdGVkLCBgciBucm93KHdvcmtlcl9jb3VudHMpYCBiZWdhbiB0aGUgYmF0dGVyeSwgYHIgc3VtKHdvcmtlcl9jb3VudHMkdGFza19jb3VudCA+PSA2MilgIGNvbXBsZXRlZCB0aGUgYmF0dGVyeSBhbmQgMTUwIHByb3ZpZGVkIGRhdGEgdGhhdCBwYXNzZWQgcWMgZm9yIGJvdGggdGltZSBwb2ludHMuIE91ciB0YXJnZXQgc2FtcGxlIHNpemUgd2FzIGRldGVybWluZWQgaW4gYWR2YW5jZSBvZiBkYXRhIGNvbGxlY3Rpb24gYW5kIGRhdGEgY29sbGVjdGlvbiBjb250aW51ZWQgdW50aWwgdGhpcyBudW1iZXIgb2YgcGFydGljaXBhbnRzIHdpdGggZGF0YSB0aGF0IHN1cnZpdmVkIHFjIHdhcyByZWFjaGVkLgoKYGBge3Igd2FybmluZz1GQUxTRSwgbWVzc2FnZT1GQUxTRX0KZGlzY19jb21wX2RhdGUgPSByZWFkLmNzdihwYXN0ZTAocmV0ZXN0X2RhdGFfcGF0aCwnTG9jYWwvZGlzY292ZXJ5X2NvbXBsZXRpb25fZGF0ZXMuY3N2JyksIGhlYWRlcj1GQUxTRSkKdmFsX2NvbXBfZGF0ZSA9IHJlYWQuY3N2KHBhc3RlMChyZXRlc3RfZGF0YV9wYXRoLCdMb2NhbC92YWxpZGF0aW9uX2NvbXBsZXRpb25fZGF0ZXMuY3N2JyksIGhlYWRlcj1GQUxTRSkKdGVzdF9jb21wX2RhdGUgPSByYmluZChkaXNjX2NvbXBfZGF0ZSwgdmFsX2NvbXBfZGF0ZSkKcm0oZGlzY19jb21wX2RhdGUsIHZhbF9jb21wX2RhdGUpCnJldGVzdF9jb21wX2RhdGUgPSByZWFkLmNzdihwYXN0ZTAocmV0ZXN0X2RhdGFfcGF0aCwnTG9jYWwvcmV0ZXN0X2NvbXBsZXRpb25fZGF0ZXMuY3N2JyksIGhlYWRlcj1GQUxTRSkKY29tcF9kYXRlcyA9IG1lcmdlKHJldGVzdF9jb21wX2RhdGUsIHRlc3RfY29tcF9kYXRlLCBieT0iVjEiKQpuYW1lcyhjb21wX2RhdGVzKSA8LSBjKCJzdWJfaWQiLCAicmV0ZXN0X2NvbXAiLCAidGVzdF9jb21wIikKY29tcF9kYXRlcyRyZXRlc3RfY29tcCA9IGFzLkRhdGUoY29tcF9kYXRlcyRyZXRlc3RfY29tcCkKY29tcF9kYXRlcyR0ZXN0X2NvbXAgPSBhcy5EYXRlKGNvbXBfZGF0ZXMkdGVzdF9jb21wKQpjb21wX2RhdGVzJGRheXNfYnR3ID0gd2l0aChjb21wX2RhdGVzLCByZXRlc3RfY29tcC10ZXN0X2NvbXApCmBgYAoKRGF0YSBjb2xsZWN0aW9uIHRvb2sgcGxhY2Ugb24gYXZlcmFnZSBgciByb3VuZChtZWFuKGFzLm51bWVyaWMoY29tcF9kYXRlcyRkYXlzX2J0dykpKWAgbnVtYmVyIG9mIGRheXMgYWZ0ZXIgdGhlIGNvbXBsZXRpb24gb2YgdGhlIGluaXRpYWwgYmF0dGVyeSB3aXRoIGEgcmFuZ2Ugb2YgYHIgcm91bmQocmFuZ2UoYXMubnVtZXJpYyhjb21wX2RhdGVzJGRheXNfYnR3KSkpWzFdYCB0byBgciByb3VuZChyYW5nZShhcy5udW1lcmljKGNvbXBfZGF0ZXMkZGF5c19idHcpKSlbMl1gIGRheXMuCgpgYGB7cn0Kcm0odGVzdF9jb21wX2RhdGUsIHJldGVzdF9jb21wX2RhdGUsIGNvbXBfZGF0ZXMpCmBgYAoKIyMgRGVtb2dyYXBoaWNzCgpgYGB7cn0KdGVzdF9kZW1vZyA8LSByZWFkLmNzdihwYXN0ZTAocmV0ZXN0X2RhdGFfcGF0aCwgJy90MV9kYXRhL2RlbW9ncmFwaGljX2hlYWx0aC5jc3YnKSkKCnJldGVzdF9kZW1vZyA8LSByZWFkLmNzdihwYXN0ZTAocmV0ZXN0X2RhdGFfcGF0aCwgJ2RlbW9ncmFwaGljX2hlYWx0aC5jc3YnKSkKCnJldGVzdF9kZW1vZyA9IHJldGVzdF9kZW1vZ1tyZXRlc3RfZGVtb2ckWCAlaW4lIHRlc3RfZGVtb2ckWCxdCgpuYW1lcyh0ZXN0X2RlbW9nKVt3aGljaChuYW1lcyh0ZXN0X2RlbW9nKSA9PSAnWCcpXSA8LSdzdWJfaWQnCm5hbWVzKHJldGVzdF9kZW1vZylbd2hpY2gobmFtZXMocmV0ZXN0X2RlbW9nKSA9PSAnWCcpXSA8LSdzdWJfaWQnCgpzdW1tYXJ5KHRlc3RfZGVtb2cgJT4lCiAgICAgICAgICBzZWxlY3QoU2V4LCBBZ2UpKQoKc3VtbWFyeShyZXRlc3RfZGVtb2cgJT4lCiAgICAgICAgICBzZWxlY3QoU2V4LCBBZ2UpKQpgYGAKCiMjIExpdGVyYXR1cmUKCk9uZSBvZiB0aGUgbWFqb3IgY29udHJpYnV0aW9ucyBvZiB0aGlzIHByb2plY3QgaXMgYSBjb21wcmVoZW5zaXZlIGxpdGVyYXR1cmUgcmV2aWV3IG9mIHRoZSByZXRlc3QgcmVsaWFiaWxpdGllcyBvZiB0aGUgc3VydmV5cyBhbmQgdGFza3MgdGhhdCB3ZXJlIHVzZWQuIFdlIHJldmlld2VkIHRoZSBsaXRlcmF0dXJlIG9uIGEgbWVhc3VyZSAoYXMgb3Bwb3NlZCB0byB0YXNrIGxldmVsKSBwYXlpbmcgYXR0ZW50aW9uIHRvIGRpZmZlcmVuY2VzIGluIHNhbXBsZSBzaXplLCB0aGUgZGVsYXkgYmV0d2VlbiB0aGUgdHdvIG1lYXN1cmVtZW50cyBhcyB3ZWxsIGFzIHRoZSBzdGF0aXN0aWMgdGhhdCB3YXMgdXNlZCB0byBhc3Nlc3MgcmVsaWFiaWxpdGllcyAoZS5nLiBTcGVhcm1hbiB2cy4gUGVhcnNvbiBjb3JyZWxhdGlvbnMpLiBIZXJlIHdlIHByZXNlbnQgYSB0YWJsZSBhbmQgYSB2aXN1YWxpemF0aW9uIHN1bW1hcml6aW5nIG91ciBmaW5kaW5ncy4gUmVmZXJlbmNlcyBtZW50aW9uZWQgaW4gdGhlIHRhYmxlIGJlbG93IGNhbiBiZSBmb3VuZCBbaGVyZV0oaHR0cHM6Ly9kb2NzLmdvb2dsZS5jb20vc3ByZWFkc2hlZXRzL2QvMWhjRUQ0X3JXU0dTRThUZDBhcVQwa1dxbExDQkF6Uk5LQlhqM3RTcmFtNTgvZWRpdD91c3A9c2hhcmluZykuICAKCmBgYHtyfQpsaXRfcmV2aWV3IDwtIHJlYWQuY3N2KCcvVXNlcnMvemV5bmVwZW5rYXZpL0Ryb3Bib3gvUG9sZHJhY2tMYWIvU1JPX1JldGVzdF9BbmFseXNlcy9pbnB1dC9saXRfcmV2aWV3X2ZpZ3VyZS5jc3YnKQoKbGl0X3JldmlldwpgYGAKCmBgYHtyIG1lc3NhZ2U9RkFMU0V9CmxpdF9yZXZpZXcgPSBsaXRfcmV2aWV3ICU+JQogIHNlcGFyYXRlKGR2LCBjKCJ0YXNrX2dyb3VwIiwgInZhciIpLCBzZXA9IlxcLiIscmVtb3ZlPUZBTFNFLGV4dHJhPSJtZXJnZSIpICU+JQogIG11dGF0ZSh0YXNrX2dyb3VwID0gZmFjdG9yKHRhc2tfZ3JvdXAsIGxldmVscyA9IHRhc2tfZ3JvdXBbb3JkZXIodGFzayldKSwKICAgICAgICAgdHlwZSA9IGFzLmNoYXJhY3Rlcih0eXBlKSkgJT4lCiAgbXV0YXRlKHRhc2tfZ3JvdXAgPSBnc3ViKCJfIiwgIiAiLCB0YXNrX2dyb3VwKSwKICAgICAgICAgdmFyID0gZ3N1YigiXyIsICIgIiwgdmFyKSkgJT4lCiAgYXJyYW5nZSh0YXNrX2dyb3VwLCByYXdfZml0LCB2YXIpICU+JQogIG11dGF0ZSh0YXNrX2dyb3VwID0gaWZlbHNlKHRhc2tfZ3JvdXAgPT0gInBzeWNob2xvZ2ljYWwgcmVmcmFjdG9yeSBwZXJpb2QgdHdvIGNob2ljZXMiLCAicHN5Y2hvbG9naWNhbCByZWZyYWN0b3J5IHBlcmlvZCIsIGlmZWxzZSh0YXNrX2dyb3VwID09ICJhbmdsaW5nIHJpc2sgdGFzayBhbHdheXMgc3VubnkiLCAiYW5nbGluZyByaXNrIHRhc2siLHRhc2tfZ3JvdXApKSkgJT4lCiAgbXV0YXRlKHRhc2tfZ3JvdXAgPSBnc3ViKCJzdXJ2ZXkiLCAiIiwgdGFza19ncm91cCkpICU+JQogIHNlbGVjdCgtbWVhc3VyZV9kZXNjcmlwdGlvbikKCgp0bXAgPSBsaXRfcmV2aWV3W2R1cGxpY2F0ZWQobGl0X3JldmlldyRyZWZlcmVuY2UpPT1GQUxTRSxdCgpucm93KHRtcCkKc3VtKHRtcCRzYW1wbGVfc2l6ZSkKbnJvdyhsaXRfcmV2aWV3KQpgYGAKCmBgYHtyfQpybSh0bXApCmBgYAoKTWVhc3VyZSBsZXZlbCBwbG90CgpgYGB7ciB3YXJuaW5nPUZBTFNFLCBtZXNzYWdlPUZBTFNFfQpsaXRfcmV2aWV3ID0gbGl0X3JldmlldyAlPiUKICBzZWxlY3QoLXJlZmVyZW5jZSkKYGBgCgpgYGB7ciB3YXJuaW5nPSBGQUxTRSwgbWVzc2FnZSA9RkFMU0UsIGVjaG89RkFMU0V9CnAxX2xlZ2VuZCA9IGxpdF9yZXZpZXcgJT4lCiAgZmlsdGVyKHRhc2sgPT0gJ3Rhc2snKSAlPiUKZ2dwbG90KGFlcyh5ID0gdmFyLCB4ID0gcmV0ZXN0X3JlbGlhYmlsaXR5KSkgKyAKICBnZW9tX3BvaW50KGFlcyhzaXplPXNhbXBsZV9zaXplLCBzaGFwZT10eXBlKSkrCiAgZmFjZXRfZ3JpZCh0YXNrX2dyb3Vwfi4sIHN3aXRjaCA9ICJ5Iiwgc2NhbGVzID0gImZyZWVfeSIsIHNwYWNlID0gImZyZWVfeSIpICsKICB0aGVtZShwYW5lbC5zcGFjaW5nID0gdW5pdCgwLjUsICJsaW5lcyIpLCAKICAgICAgICBzdHJpcC5wbGFjZW1lbnQgPSAib3V0c2lkZSIsCiAgICAgICAgc3RyaXAudGV4dC55ID0gZWxlbWVudF90ZXh0KGFuZ2xlPTE4MCksCiAgICAgICAgcGFuZWwuYmFja2dyb3VuZCA9IGVsZW1lbnRfcmVjdChmaWxsID0gTkEpLAogICAgICAgIHBhbmVsLmdyaWQubWFqb3IgPSBlbGVtZW50X2xpbmUoY29sb3VyID0gImdyZXk4MCIpLAogICAgICAgIGxlZ2VuZC5wb3NpdGlvbiA9ICdib3R0b20nKSArIAogIHhsYWIoIiIpKwogIHlsYWIoIiIpKwogIHNjYWxlX3hfY29udGludW91cyhsaW1pdHMgPSBjKC0wLjI1LDEpLCBicmVha3M9YygtMC4yNSwgMCwgMC4yNSwgMC41LCAwLjc1LCAxKSkrCiAgc2NhbGVfc2hhcGVfbWFudWFsKGJyZWFrcyA9IHNvcnQobGl0X3JldmlldyR0eXBlKSwgdmFsdWVzID0gYygxNSwgMTYsIDE3LCAzKSkKCnAxID0gbGl0X3JldmlldyAlPiUKICBmaWx0ZXIodGFzayA9PSAndGFzaycpICU+JQpnZ3Bsb3QoYWVzKHkgPSB2YXIsIHggPSByZXRlc3RfcmVsaWFiaWxpdHkpKSArIAogIGdlb21fcG9pbnQoYWVzKHNpemU9c2FtcGxlX3NpemUsIHNoYXBlID0gdHlwZSksIGNvbG9yPScjMDBCRkM0JykrCiAgZmFjZXRfZ3JpZCh0YXNrX2dyb3Vwfi4sIHN3aXRjaCA9ICJ5Iiwgc2NhbGVzID0gImZyZWVfeSIsIHNwYWNlID0gImZyZWVfeSIpICsKICB0aGVtZShwYW5lbC5zcGFjaW5nID0gdW5pdCgwLjUsICJsaW5lcyIpLCAKICAgICAgICBzdHJpcC5wbGFjZW1lbnQgPSAib3V0c2lkZSIsCiAgICAgICAgc3RyaXAudGV4dC55ID0gZWxlbWVudF90ZXh0KGFuZ2xlPTE4MCksCiAgICAgICAgcGFuZWwuYmFja2dyb3VuZCA9IGVsZW1lbnRfcmVjdChmaWxsID0gTkEpLAogICAgICAgIHBhbmVsLmdyaWQubWFqb3IgPSBlbGVtZW50X2xpbmUoY29sb3VyID0gImdyZXk4MCIpLAogICAgICAgIGxlZ2VuZC5wb3NpdGlvbiA9ICdib3R0b20nKSArIAogIHhsYWIoIiIpKwogIHlsYWIoIiIpKwogIHNjYWxlX3hfY29udGludW91cyhsaW1pdHMgPSBjKC0wLjI1LDEpLCBicmVha3M9YygtMC4yNSwgMCwgMC4yNSwgMC41LCAwLjc1LCAxKSkrCiAgc2NhbGVfc2hhcGVfbWFudWFsKGJyZWFrcyA9IHNvcnQobGl0X3JldmlldyR0eXBlKSwgdmFsdWVzID0gYygxNSwgMTYsIDE3LCAzKSkKCnAyID0gbGl0X3JldmlldyAlPiUKICBmaWx0ZXIodGFzayA9PSAnc3VydmV5JykgJT4lCmdncGxvdChhZXMoeSA9IHZhciwgeCA9IHJldGVzdF9yZWxpYWJpbGl0eSkpICsgCiAgZ2VvbV9wb2ludChhZXMoc2l6ZT1zYW1wbGVfc2l6ZSwgc2hhcGUgPSB0eXBlKSwgY29sb3IgPSAnI0Y4NzY2RCcpKwogIGZhY2V0X2dyaWQodGFza19ncm91cH4uLCBzd2l0Y2ggPSAieSIsIHNjYWxlcyA9ICJmcmVlX3kiLCBzcGFjZSA9ICJmcmVlX3kiKSArCiAgdGhlbWUocGFuZWwuc3BhY2luZyA9IHVuaXQoMC41LCAibGluZXMiKSwgCiAgICAgICAgc3RyaXAucGxhY2VtZW50ID0gIm91dHNpZGUiLAogICAgICAgIHN0cmlwLnRleHQueSA9IGVsZW1lbnRfdGV4dChhbmdsZT0xODApLAogICAgICAgIHBhbmVsLmJhY2tncm91bmQgPSBlbGVtZW50X3JlY3QoZmlsbCA9IE5BKSwKICAgICAgICBwYW5lbC5ncmlkLm1ham9yID0gZWxlbWVudF9saW5lKGNvbG91ciA9ICJncmV5ODAiKSkgKyAKICB4bGFiKCIiKSsKICB5bGFiKCIiKSsKICBzY2FsZV94X2NvbnRpbnVvdXMobGltaXRzID0gYygtMC4yNSwxKSwgYnJlYWtzPWMoLTAuMjUsIDAsIDAuMjUsIDAuNSwgMC43NSwgMSkpKwogIHNjYWxlX3NoYXBlX21hbnVhbChicmVha3MgPSBzb3J0KGxpdF9yZXZpZXckdHlwZSksIHZhbHVlcyA9IGMoMTUsIDE3LCAzKSkKCm15bGVnZW5kPC1nX2xlZ2VuZChwMV9sZWdlbmQpCgpwMyA8LSBhcnJhbmdlR3JvYihhcnJhbmdlR3JvYihwMSArdGhlbWUobGVnZW5kLnBvc2l0aW9uPSJub25lIiksCiAgICAgICAgICAgICAgICAgICAgICAgICBwMiArIHRoZW1lKGxlZ2VuZC5wb3NpdGlvbj0ibm9uZSIpLAogICAgICAgICAgICAgICAgICAgICAgICAgbnJvdz0xKSwKICAgICAgICAgICAgIG15bGVnZW5kLCBucm93PTIsaGVpZ2h0cz1jKDEwLCAxKSkKCmdnc2F2ZSgnTGl0X1Jldmlld19QbG90LmpwZycsIHBsb3QgPSBwMywgZGV2aWNlID0gImpwZWciLCBwYXRoID0gIi9Vc2Vycy96ZXluZXBlbmthdmkvRHJvcGJveC9Qb2xkcmFja0xhYi9TUk9fUmV0ZXN0X0FuYWx5c2VzL291dHB1dC9maWd1cmVzLyIsIHdpZHRoID0gMjQsIGhlaWdodCA9IDIwLCB1bml0cyA9ICJpbiIsIGRwaT0xMDApCnJtKHAxLCBwMiwgcDMsIHAxX2xlZ2VuZCwgbXlsZWdlbmQpCmBgYAoKYGBge3IgZWNobz1GQUxTRSwgb3V0LndpZHRoPScxMDAlJ30Ka25pdHI6OmluY2x1ZGVfZ3JhcGhpY3MoJy9Vc2Vycy96ZXluZXBlbmthdmkvRHJvcGJveC9Qb2xkcmFja0xhYi9TUk9fUmV0ZXN0X0FuYWx5c2VzL291dHB1dC9maWd1cmVzL0xpdF9SZXZpZXdfUGxvdC5qcGcnKQpgYGAKCkJlY2F1c2UgdGhpcyBwbG90IGlzIGRpZmZpY3VsdCB0byBkaWdlc3Qgd2Ugc3VtbWFyaXplIGl0IG9uIGEgdGFzayBsZXZlbCB0byBnaXZlIGEgZ2VuZXJhbCBzZW5zZSBvZiB0aGUgbWFpbiB0YWtlYXdheXMuIFRoaXMgcGxvdCBuYXR1cmFsbHkgZGlzcmVnYXJkcyBtdWNoIG9mIHRoZSBmaW5lIGdyYWluZWQgaW5mb3JtYXRpb24uCgpgYGB7cn0KcDFfdF9sZWdlbmQgPC0gbGl0X3JldmlldyAlPiUKICBmaWx0ZXIodGFzayA9PSAndGFzaycpICU+JQpnZ3Bsb3QoYWVzKHkgPSBmYWN0b3IodGFza19ncm91cCwgbGV2ZWxzPXJldih1bmlxdWUodGFza19ncm91cCkpKSwgeCA9IHJldGVzdF9yZWxpYWJpbGl0eSkpICsgCiAgZ2VvbV9wb2ludChhZXMoc2l6ZT1zYW1wbGVfc2l6ZSwgc2hhcGUgPSB0eXBlKSwgY29sb3I9J2JsYWNrJykrCiAgdGhlbWUoYXhpcy50ZXh0LnkgPSBlbGVtZW50X3RleHQoc2l6ZT00MyksCiAgICAgICAgbGVnZW5kLnBvc2l0aW9uID0gJ2JvdHRvbScsCiAgICAgICAgYXhpcy50ZXh0LnggPSBlbGVtZW50X3RleHQoc2l6ZT0yMyksCiAgICAgICAgYXhpcy50aXRsZS54ID0gZWxlbWVudF90ZXh0KHNpemU9MzApLCAKICAgICAgICBsZWdlbmQudGV4dCA9IGVsZW1lbnRfdGV4dChzaXplPTIwKSwgCiAgICAgICAgbGVnZW5kLmtleS53aWR0aCA9IHVuaXQoMC43NSwgImluY2hlcyIpLCAKICAgICAgICBsZWdlbmQudGl0bGUgPSBlbGVtZW50X3RleHQoc2l6ZT0yOCksCiAgICAgICAgbGVnZW5kLnNwYWNpbmcueCA9IHVuaXQoMC41LCAiaW5jaGVzIikpICsgCiAgZ3VpZGVzKHNpemUgPSBndWlkZV9sZWdlbmQob3ZlcnJpZGUuYWVzID0gbGlzdChzaXplPWMoOSwxOCwyOCkpKSwKICAgICAgICAgc2hhcGUgPSBndWlkZV9sZWdlbmQob3ZlcnJpZGUuYWVzID0gbGlzdChzaXplPTE4KSkpKwogIHhsYWIoIlJldGVzdCBSZWxpYWJpbGl0eSIpKwogIHlsYWIoIiIpKwogIHNjYWxlX3hfY29udGludW91cyhsaW1pdHMgPSBjKC0wLjI1LDEpLCBicmVha3M9YygtMC4yNSwgMCwgMC4yNSwgMC41LCAwLjc1LCAxKSkrCiAgc2NhbGVfeV9kaXNjcmV0ZShsYWJlbHMgPSBmdW5jdGlvbih4KSBzdHJfd3JhcCh4LCB3aWR0aCA9IDEwKSkrCiAgc2NhbGVfc2hhcGVfbWFudWFsKGJyZWFrcyA9IHNvcnQobGl0X3JldmlldyR0eXBlKSwgdmFsdWVzID0gYygxNSwgMTYsIDE3LCAzKSwgbmFtZT0iVHlwZSIpKwogIHNjYWxlX3NpemVfY29udGludW91cyhuYW1lID0gIlNhbXBsZSBTaXplIikrCiAgZ2VvbV92bGluZSh4aW50ZXJjZXB0ID0gMCwgY29sb3IgPSAicmVkIiwgc2l6ZSA9IDEpCgpwMV90IDwtIGxpdF9yZXZpZXcgJT4lCiAgZmlsdGVyKHRhc2sgPT0gJ3Rhc2snKSAlPiUKZ2dwbG90KGFlcyh5ID0gZmFjdG9yKHRhc2tfZ3JvdXAsIGxldmVscz1yZXYodW5pcXVlKHRhc2tfZ3JvdXApKSksIHggPSByZXRlc3RfcmVsaWFiaWxpdHkpKSArIAogIGdlb21fcG9pbnQoYWVzKHNpemU9c2FtcGxlX3NpemUsIHNoYXBlID0gdHlwZSksIGNvbG9yPScjMDBCRkM0JykrCiAgdGhlbWUoYXhpcy50ZXh0LnkgPSBlbGVtZW50X3RleHQoc2l6ZT00MyksCiAgICAgICAgbGVnZW5kLnBvc2l0aW9uID0gJ25vbmUnLAogICAgICAgIGF4aXMudGV4dC54ID0gZWxlbWVudF90ZXh0KHNpemU9MjMpLAogICAgICAgIGF4aXMudGl0bGUueCA9IGVsZW1lbnRfdGV4dChzaXplPTMwKSkgKyAKICB4bGFiKCJSZXRlc3QgUmVsaWFiaWxpdHkiKSsKICB5bGFiKCIiKSsKICBzY2FsZV94X2NvbnRpbnVvdXMobGltaXRzID0gYygtMC4yNSwxKSwgYnJlYWtzPWMoLTAuMjUsIDAsIDAuMjUsIDAuNSwgMC43NSwgMSkpKwogIHNjYWxlX3lfZGlzY3JldGUobGFiZWxzID0gZnVuY3Rpb24oeCkgc3RyX3dyYXAoeCwgd2lkdGggPSAxMCkpKwogIHNjYWxlX3NoYXBlX21hbnVhbChicmVha3MgPSBzb3J0KGxpdF9yZXZpZXckdHlwZSksIHZhbHVlcyA9IGMoMTUsIDE2LCAxNywgMykpKwogIHNjYWxlX3NpemVfY29udGludW91cyhyYW5nZSA9IGMoNSwgMzUpKSsKICBnZW9tX3ZsaW5lKHhpbnRlcmNlcHQgPSAwLCBjb2xvciA9ICJyZWQiLCBzaXplID0gMSkKCnAyX3QgPC0gbGl0X3JldmlldyAlPiUKICBmaWx0ZXIodGFzayA9PSAnc3VydmV5JykgJT4lCmdncGxvdChhZXMoeSA9IGZhY3Rvcih0YXNrX2dyb3VwLCBsZXZlbHM9cmV2KHVuaXF1ZSh0YXNrX2dyb3VwKSkpLCB4ID0gcmV0ZXN0X3JlbGlhYmlsaXR5KSkgKyAKICBnZW9tX3BvaW50KGFlcyhzaXplPXNhbXBsZV9zaXplLCBzaGFwZSA9IHR5cGUpLCBjb2xvcj0nI0Y4NzY2RCcpKwogIHRoZW1lKGF4aXMudGV4dC55ID0gZWxlbWVudF90ZXh0KHNpemU9NDMpLAogICAgICAgIGxlZ2VuZC5wb3NpdGlvbiA9ICdub25lJywKICAgICAgICBheGlzLnRleHQueCA9IGVsZW1lbnRfdGV4dChzaXplPTIzKSwKICAgICAgICBheGlzLnRpdGxlLnggPSBlbGVtZW50X3RleHQoc2l6ZT0zMCkpICsgCiAgeGxhYigiUmV0ZXN0IFJlbGlhYmlsaXR5IikrCiAgeWxhYigiIikrCiAgc2NhbGVfeF9jb250aW51b3VzKGxpbWl0cyA9IGMoLTAuMjUsMSksIGJyZWFrcz1jKC0wLjI1LCAwLCAwLjI1LCAwLjUsIDAuNzUsIDEpKSsKICBzY2FsZV95X2Rpc2NyZXRlKGxhYmVscyA9IGZ1bmN0aW9uKHgpIHN0cl93cmFwKHgsIHdpZHRoID0gMTApKSsKICBzY2FsZV9zaGFwZV9tYW51YWwoYnJlYWtzID0gc29ydChsaXRfcmV2aWV3JHR5cGUpLCB2YWx1ZXMgPSBjKDE1LCAxNiwgMTcsIDMpKSsKICBzY2FsZV9zaXplX2NvbnRpbnVvdXMocmFuZ2UgPSBjKDUsIDM1KSkrCiAgZ2VvbV92bGluZSh4aW50ZXJjZXB0ID0gMCwgY29sb3IgPSAicmVkIiwgc2l6ZSA9IDEpCgpteWxlZ2VuZDwtZ19sZWdlbmQocDFfdF9sZWdlbmQpCgpwM190IDwtIGFycmFuZ2VHcm9iKGFycmFuZ2VHcm9iKHAxX3QsIHAyX3QsIG5yb3c9MSksIG15bGVnZW5kLCBucm93PTIsaGVpZ2h0cz1jKDEwLCAxKSkKCmdnc2F2ZSgnTGl0X1Jldmlld19QbG90X3QuanBnJywgcGxvdCA9IHAzX3QsIGRldmljZSA9ICJqcGVnIiwgcGF0aCA9ICIvVXNlcnMvemV5bmVwZW5rYXZpL0Ryb3Bib3gvUG9sZHJhY2tMYWIvU1JPX1JldGVzdF9BbmFseXNlcy9vdXRwdXQvZmlndXJlcy8iLCB3aWR0aCA9IDI3LCBoZWlnaHQgPSA0OCwgdW5pdHMgPSAiaW4iLCBsaW1pdHNpemUgPSBGQUxTRSwgZHBpID0gNzIpCnJtKHAxX3QsIHAyX3QsIHAzX3QsIG15bGVnZW5kLCBwMV90X2xlZ2VuZCkKYGBgCgpgYGB7ciBlY2hvPUZBTFNFLCBvdXQud2lkdGg9JzEwMCUnfQprbml0cjo6aW5jbHVkZV9ncmFwaGljcygnL1VzZXJzL3pleW5lcGVua2F2aS9Ecm9wYm94L1BvbGRyYWNrTGFiL1NST19SZXRlc3RfQW5hbHlzZXMvb3V0cHV0L2ZpZ3VyZXMvTGl0X1Jldmlld19QbG90X3QuanBnJykKYGBgCgpBbiBpbnRlcmFjdGl2ZSB2ZXJzaW9uIG9mIHRoaXMgcGxvdCBjb3VsZCBiZSBmaW5kIFtoZXJlXShodHRwczovL3plbmthdmkuZ2l0aHViLmlvL1NST19SZXRlc3RfQW5hbHlzZXMvb3V0cHV0L3JlcG9ydHMvTGl0X1Jldmlld19GaWd1cmUuaHRtbCkKClRha2Vhd2F5cyBmcm9tIHRoaXMgcmV2aWV3IGFyZTogICAKLSBTdXJ2ZXkgbWVhc3VyZXMgaGF2ZSBiZWVuIHJlcG9ydGVkIHRvIGhpZ2hlciByZWxpYWJpbGl0eSBjb21wYXJlZCB0byB0YXNrIG1lYXN1cmVzICAKLSBTdXJ2ZXkgbWVhc3VyZXMgaGF2ZSBsZXNzIHZhcmlhYmlsaXR5IGluIHRoZSByZXBvcnRlZCByZWxpYWJpbHRpeSBlc3RpbWF0ZXMgY29tcGFyZWQgdG8gdGFzayBtZWFzdXJlcyAgIAoKIyMgTG9hZGluZyBkYXRhc2V0cwoKVGhlIHZhcmlhYmxlcyBpbmNsdWRlZCBpbiB0aGlzIHJlcG9ydCBhcmU6ICAKLSBtZWFuaW5nZnVsIHZhcmlhYmxlcyAoaW5jbHVkZXMgb25seSBoZGRkbSBwYXJhbWV0ZXJzKSAgCi0gRVogZGlmZnVzaW9uIHBhcmFtZXRlcnMgIAotIFJhdyBSVCBhbmQgQWNjdXJhY3kgbWVhc3VyZXMgIAotIFZhcmlhYmxlcyBmb3VuZCBpbiB0aGUgbGl0ZXJhdHVyZSAoZm9yIGNvbXBhcmlzb24pICAKCmBgYHtyIGVjaG89RkFMU0V9CiNHZXQgdmFyaWFibGVzIG9mIGludGVyZXN0IGZyb20gSWFuJ3MgcmVsZWFzZQp0bXAxIDwtIHJlYWQuY3N2KHBhc3RlMCh0ZXN0X2RhdGFfcGF0aCwnbWVhbmluZ2Z1bF92YXJpYWJsZXMuY3N2JykpCnRtcDIgPC0gcmVhZC5jc3YocGFzdGUwKHRlc3RfZGF0YV9wYXRoLCdtZWFuaW5nZnVsX3ZhcmlhYmxlc19ub0RETS5jc3YnKSkKdG1wMyA8LSByZWFkLmNzdihwYXN0ZTAodGVzdF9kYXRhX3BhdGgsJ21lYW5pbmdmdWxfdmFyaWFibGVzX0VaLmNzdicpKQpyZXRlc3RfcmVwb3J0X3ZhcnMgPSBjKG5hbWVzKHRtcDEpLCBuYW1lcyh0bXAyKSwgbmFtZXModG1wMykpCnJldGVzdF9yZXBvcnRfdmFycyA9IHVuaXF1ZShyZXRlc3RfcmVwb3J0X3ZhcnMpCmxpdF9yZXZfdmFycyA9IGFzLmNoYXJhY3Rlcih1bmlxdWUobGl0X3JldmlldyRkdilbd2hpY2godW5pcXVlKGxpdF9yZXZpZXckZHYpICVpbiUgcmV0ZXN0X3JlcG9ydF92YXJzID09IEZBTFNFKV0pCnJldGVzdF9yZXBvcnRfdmFycyA9IGMocmV0ZXN0X3JlcG9ydF92YXJzLCBsaXRfcmV2X3ZhcnMpCnJtKHRtcDEsIHRtcDIsIHRtcDMsIGxpdF9yZXZfdmFycykKYGBgCgojIyMgTG9hZCB0aW1lIDEgZGF0YQpgYGB7cn0KdGVzdF9kYXRhIDwtIHJlYWQuY3N2KHBhc3RlMChyZXRlc3RfZGF0YV9wYXRoLCd0MV9kYXRhL3ZhcmlhYmxlc19leGhhdXN0aXZlLmNzdicpKQoKdGVzdF9kYXRhIDwtIHRlc3RfZGF0YVssbmFtZXModGVzdF9kYXRhKSAlaW4lIHJldGVzdF9yZXBvcnRfdmFyc10KCnRlc3RfZGF0YSRYIDwtIGFzLmNoYXJhY3Rlcih0ZXN0X2RhdGEkWCkKbmFtZXModGVzdF9kYXRhKVt3aGljaChuYW1lcyh0ZXN0X2RhdGEpID09ICdYJyldIDwtJ3N1Yl9pZCcgCmBgYAoKRm9yIHJlZmVyZW5jZSBoZXJlIGFyZSB0aGUgdmFyaWFibGVzIHRoYXQgYXJlICoqbm90KiogaW5jbHVkZWQgaW4gdGhlIGFuYWx5c2VzIG9mIHRoZSByZW1haW5kZXIgb2YgdGhpcyByZXBvcnQgYmVjYXVzZSB0aGV5IHdlcmUgbm90IG9mIHRoZW9yZXRpY2FsIGludGVyZXN0IGluIGZhY3RvciBzdHJ1Y3R1cmUgYW5hbHlzZXMgb2YgdGhpcyBkYXRhIHNvIGZhci4gVGhlc2UgaW5jbHVkZSBkcmlmdCBkaWZmdXNpb24gYW5kIG90aGVyIG1vZGVsIHBhcmFtZXRlcnMgZm9yIHNwZWNpZmljIGNvbmRpdGlvbnMgd2l0aGluIGEgdGFzazsgc3VydmV5IHZhcmlhYmxlcyB0aGF0IGFyZSBub3QgcGFydCBvZiB0aGUgZGVwZW5kYW50IHZhcmlhYmxlcyBmb3IgdGhhdCBzdXJ2ZXkgaW4gdGhlIGxpdGVyYXR1cmUgYW5kIGRlbW9ncmFwaGljcyAodGhlc2UgYXJlIHNhdmVkIGZvciBwcmVkaWN0aW9uIGFuYWx5c2VzKS4KCmBgYHtyfQp0ZXN0X2RhdGEyIDwtIHJlYWQuY3N2KHBhc3RlMChyZXRlc3RfZGF0YV9wYXRoLCAndDFfZGF0YS92YXJpYWJsZXNfZXhoYXVzdGl2ZS5jc3YnKSkKCmRmIDwtIGRhdGEuZnJhbWUobmFtZXModGVzdF9kYXRhMilbd2hpY2gobmFtZXModGVzdF9kYXRhMikgJWluJSBuYW1lcyh0ZXN0X2RhdGEpID09IEZBTFNFKV0pCm5hbWVzKGRmKSA9IGMoJ3ZhcnMnKQoKZGYKYGBgCgpgYGB7ciBlY2hvPUZBTFNFfQpybSh0ZXN0X2RhdGEyLCBkZikKYGBgCgojIyMgTG9hZCB0aW1lIDIgZGF0YSAKYGBge3J9CnJldGVzdF9kYXRhIDwtIHJlYWQuY3N2KHBhc3RlMChyZXRlc3RfZGF0YV9wYXRoLCd2YXJpYWJsZXNfZXhoYXVzdGl2ZS5jc3YnKSkKCnJldGVzdF9kYXRhIDwtIHJldGVzdF9kYXRhWyxuYW1lcyhyZXRlc3RfZGF0YSkgJWluJSByZXRlc3RfcmVwb3J0X3ZhcnNdCgpyZXRlc3RfZGF0YSRYIDwtIGFzLmNoYXJhY3RlcihyZXRlc3RfZGF0YSRYKQpuYW1lcyhyZXRlc3RfZGF0YSlbd2hpY2gobmFtZXMocmV0ZXN0X2RhdGEpID09ICdYJyldIDwtJ3N1Yl9pZCcgCnJldGVzdF9kYXRhID0gcmV0ZXN0X2RhdGFbcmV0ZXN0X2RhdGEkc3ViX2lkICVpbiUgdGVzdF9kYXRhJHN1Yl9pZCxdCmBgYAoKIyMjIFJlcGxhY2UgSERETSBwYXJhbWV0ZXJzIGluIHQxIGRhdGEKClNpbmNlIEhERE0gcGFyYW1ldGVycyBkZXBlbmQgb24gdGhlIHNhbXBsZSBvbiB3aGljaCB0aGV5IGFyZSBmaXQgd2UgcmVmaXQgdGhlIG1vZGVsIG9uIHQxIGRhdGEgZm9yIHRoZSBzdWJqZWN0cyB0aGF0IGhhdmUgdDIgZGF0YS4gSGVyZSB3ZSByZXBsYWNlIHRoZSBIRERNIHBhcmFtZXRlcnMgaW4gdGhlIGN1cnJlbnQgdDEgZGF0YXNldCB3aXRoIHRoZXNlIHJlZml0dGVkIHZhbHVlcy4gCgpgYGB7cn0KaGRkbV9yZWZpdHMgPC0gcmVhZC5jc3YocGFzdGUwKHJldGVzdF9kYXRhX3BhdGgsJ3QxX2RhdGEvaGRkbV9yZWZpdHNfZXhoYXVzdGl2ZS5jc3YnKSkKCmhkZG1fcmVmaXRzID0gaGRkbV9yZWZpdHNbLG5hbWVzKGhkZG1fcmVmaXRzKSAlaW4lIHJldGVzdF9yZXBvcnRfdmFyc10KCmhkZG1fcmVmaXRzJFggPC0gYXMuY2hhcmFjdGVyKGhkZG1fcmVmaXRzJFgpCm5hbWVzKGhkZG1fcmVmaXRzKVt3aGljaChuYW1lcyhoZGRtX3JlZml0cykgPT0gJ1gnKV0gPC0nc3ViX2lkJyAKCiNGb3IgbGF0ZXIgY29tcGFyaXNvbiBvZiB3aGV0aGVyIGZpdHRpbmcgdGhlIERETSBwYXJhbWV0ZXJzIG9uIGZ1bGwgb3IgcmV0ZXN0IHNhbXBsZSBtYWtlcyBhIGJpZyBkaWZmZXJlbmNlCnRlc3RfZGF0YV9mdWxsX3NhbXBsZV9oZGRtIDwtIHRlc3RfZGF0YQoKI2Ryb3AgaGRkbSBjb2x1bW5zIGZyb20gdGVzdF9kYXRhCnRlc3RfZGF0YSA9IGNiaW5kKHRlc3RfZGF0YSRzdWJfaWQsIHRlc3RfZGF0YVssbmFtZXModGVzdF9kYXRhKSAlaW4lIG5hbWVzKGhkZG1fcmVmaXRzKSA9PSBGQUxTRV0pCgojZml4IG5hbWluZyBiZWZvcmUgbWVyZ2luZwpuYW1lcyh0ZXN0X2RhdGEpW3doaWNoKG5hbWVzKHRlc3RfZGF0YSkgPT0gJ3Rlc3RfZGF0YSRzdWJfaWQnKV0gPC0nc3ViX2lkJwoKI21lcmdlIGhkZG0gcmVmaXRzIHRvIHRlc3QgZGF0YQp0ZXN0X2RhdGEgPSBtZXJnZSh0ZXN0X2RhdGEsIGhkZG1fcmVmaXRzLCBieT0ic3ViX2lkIikKYGBgCgojIFJlc3VsdHMKCiMjIERhdGEgcXVhbGl0eSBjaGVja3MKCiMjIyBEZW1vZ3JhcGhpY3MgcmVsaWFiaWxpdHkKClBvaW50IGVzdGltYXRlcyBvZiByZWxpYWJpbGl0eSBmb3IgdGhlIGRlbW9ncmFwaGljIHZhcmlhYmVscy4KCmBgYHtyfQpudW1lcmljX2NvbHMgPSBjKCkKCmZvcihpIGluIDE6bGVuZ3RoKG5hbWVzKHRlc3RfZGVtb2cpKSl7CiAgaWYoaXMubnVtZXJpYyh0ZXN0X2RlbW9nWyxpXSkpewogICAgbnVtZXJpY19jb2xzIDwtIGMobnVtZXJpY19jb2xzLCBuYW1lcyh0ZXN0X2RlbW9nKVtpXSkKICB9Cn0KCmRlbW9nX3JlbF9kZiA8LSBkYXRhLmZyYW1lKHNwZWFybWFuID0gcmVwKE5BLCBsZW5ndGgobnVtZXJpY19jb2xzKSksCiAgICAgICAgICAgICAgICAgICAgIGljYyA9IHJlcChOQSwgbGVuZ3RoKG51bWVyaWNfY29scykpLAogICAgICAgICAgICAgICAgICAgICBwZWFyc29uID0gcmVwKE5BLCBsZW5ndGgobnVtZXJpY19jb2xzKSkpCgpyb3cubmFtZXMoZGVtb2dfcmVsX2RmKSA8LSBudW1lcmljX2NvbHMKCmZvcihpIGluIDE6bGVuZ3RoKG51bWVyaWNfY29scykpewogIGRlbW9nX3JlbF9kZltudW1lcmljX2NvbHNbaV0sICdzcGVhcm1hbiddIDwtIGdldF9zcGVhcm1hbihudW1lcmljX2NvbHNbaV0sIHQxX2RmID0gdGVzdF9kZW1vZywgdDJfZGYgPSByZXRlc3RfZGVtb2cpIAogIGRlbW9nX3JlbF9kZltudW1lcmljX2NvbHNbaV0sICdpY2MnXSA8LSBnZXRfaWNjKG51bWVyaWNfY29sc1tpXSwgdDFfZGYgPSB0ZXN0X2RlbW9nLCB0Ml9kZiA9IHJldGVzdF9kZW1vZykKICBkZW1vZ19yZWxfZGZbbnVtZXJpY19jb2xzW2ldLCAncGVhcnNvbiddIDwtIGdldF9wZWFyc29uKG51bWVyaWNfY29sc1tpXSwgdDFfZGYgPSB0ZXN0X2RlbW9nLCB0Ml9kZiA9IHJldGVzdF9kZW1vZykKfQoKZGVtb2dfcmVsX2RmID0gZGVtb2dfcmVsX2RmICU+JQogIG11dGF0ZSh2YXIgPSByb3cubmFtZXMoLikpICU+JQogIHNlbGVjdCh2YXIsIGljYywgc3BlYXJtYW4sIHBlYXJzb24pICU+JQogIGFycmFuZ2UoLWljYykKCmRlbW9nX3JlbF9kZgpgYGAKCmBgYHtyfQpzanQuZGYoZGVtb2dfcmVsX2RmICU+JSBtdXRhdGVfaWYoaXMubnVtZXJpYywgZnVucyhyb3VuZCguLCAzKSkpLCBkZXNjcmliZT1GLCBoaWRlLnByb2dyZXNzID0gVFJVRSwgc2hvdy5yb3duYW1lcyA9IEZBTFNFLCBmaWxlID0gIi9Vc2Vycy96ZXluZXBlbmthdmkvRHJvcGJveC9Qb2xkcmFja0xhYi9TUk9fUmV0ZXN0X0FuYWx5c2VzL291dHB1dC90YWJsZXMvZGVtb2dfcmVsX3RhYmxlLmRvYyIpCmBgYAoKCmBgYHtyfQpzdW1tYXJ5KGRlbW9nX3JlbF9kZikKYGBgCgpgYGB7ciBlY2hvPUZBTFNFfQpybSh0ZXN0X2RlbW9nLCByZXRlc3RfZGVtb2csIGRlbW9nX3JlbF9kZiwgbnVtZXJpY19jb2xzKQpgYGAKCiMjIyBFZmZlY3Qgb2YgZGVsYXlzIGJldHdlZW4gdGhlIHR3byBtZWFzdXJlbWVudHMKCkR1ZSB0byBvdXIgZGF0YSBjb2xsZWN0aW9uIHN0cmF0ZWd5IHdlIGRpZCBub3Qgc3RyaWN0bHkgY29udHJvbCBmb3IgdGhlIGRlbGF5IGJldHdlZW4gdGhlIHR3byBtZWFzdXJlbWVudHMgYXMgc3RhbmRhcmQgcHN5Y2hvbWV0cmljcyBzdHVkaWVzIG1lYXN1cmluZyByZXRlc3QgcmVsaWFiaWxpdHkgbWlnaHQuIAoKQW4gaW5kaXZpZHVhbCBkaWZmZXJlbmNlIG1lYXN1cmUgd291bGQgcHJlZmVyYWJseSByZW1haW4gc3RhYmxlIHJlZ2FyZGxlc3Mgb2YgdGhlIGRlbGF5IGJldHdlZW4gbXVsdGlwbGUgbWVhc3VyZW1lbnRzLgoKU2luY2Ugd2Ugb25seSBoYWQgdHdvIG1lYXN1cmVtZW50cyB3ZSBjb3VsZCBub3QgdGVzdCBkaXJlY3RseSB3aGV0aGVyIGEgbWVhc3VyZSBiZWNvbWVzIGxlc3MgcmVsaWFibGUgZGVwZW5kaW5nIG9uIHRoZSBkZWxheSBiZXR3ZWVuIHRoZSB0d28gdGltZSBwb2ludHMgKFRoZSBhdmVyYWdlIG51bWJlciBvZiBkYXlzIGJldHdlZW4gdHdvIG1lYXN1cmVtZW50cyB3b3VsZCBiZSB0aGUgc2FtZSBmb3IgYWxsIG1lYXN1cmVzIHNpbmNlIHJlbGlhYmlsaXR5IGlzIGEgbWVhc3VyZSBsZXZlbCBtZXRyaWMgd2hpbGUgZGF5cyBiZXR3ZWVuIGNvbXBsZXRpb24gYSBzdWJqZWN0IGxldmVsIG9uZSkuCgpTbyBpZiB5b3UgcmVncmVzcyB0aGUgYXZlcmFnZSBkaWZmZXJlbmNlIGZvciBlYWNoIG1lYXN1cmUgb24gYXZlcmFnZSBkZWxheSBiZXR3ZWVuIHR3byBtZWFzdXJlbWVudHMgeW91IGFyZSByZWdyZXNzaW5nIGEgdmVjdG9yIHdpdGggdmFyeWluZyBudW1iZXJzIG9uIGEgdmVjdG9yIG9mIHNhbWUgdmFsdWVzLCBsaWtlIGEgdCB0ZXN0IGFza2luZyBpZiB0aGUgbWVhbiBvZiB0aGUgdmFyeWluZyBjb2x1bW4sIGluIHRoaXMgY2FzZSB0aGUgYXZlcmFnZSBkaWZmZXJlbmNlIHNjb3JlLCBpcyBkaWZmZXJlbnQgdGhhbiB0aGUgdW5pcXVlIHZhbHVlIGluIHRoZSBzaW5nbGUgdmFsdWUgY29sdW1uLCBpLmUuIHRoZSBhdmVyYWdlIGRlbGF5IGJldHdlZW4gdHdvIHRpbWUgcG9pbnRzIChOb3RlIHRoYXQgdGhpcyBzaG91bGQgYmUgdHJ1ZSBpbiB0aGVvcnkgYnV0IGluIHByYWN0aWNlIHNpbmNlIGZvciBlYWNoIG1lYXN1cmUgdGhlcmUgbWlnaHQgc3ViamVjdHMgZm9yIHdob20gdGhlIGR2IGNvdWxkIG5vdCBiZSBjYWxjdWxhdGVkIHRoZXJlIG1pZ2h0IGJlID4xIHVuaXF1ZSB2YWx1ZXMgZm9yIHRoZSBhdmVyYWdlIGRlbGF5IGJldHdlZW4gdGhlIHR3byB0aW1lIHBvaW50cykuIFRoaXMgYW5hbHlzaXMgaXMgbm90IG1lYW5pbmdmdWwuIAoKSW5zdGVhZCBvZiB0aGUgc3VtbWFyeSBtZXRyaWMgbGlrZSB0aGUgcmV0ZXN0IHJlbGlhYmlsaXR5IGVzdGltYXRlIGZvciBlYWNoIG1lYXN1cmUgd2UgY2FuIGNoZWNrIHdoZXRoZXIgdGhlIGRpZmZlcmVuY2Ugc2NvcmUgZGlzdHJpYnV0aW9uIGZvciBlYWNoIG1lYXN1cmUgZGVwZW5kcyBvbiB0aGUgZGVsYXkgYmV0d2VlbiB0aGUgdHdvIG1lYXN1cmVtZW50cy4gU2luY2UgdGhlIGRpZmZlcmVuY2Ugc2NvcmUgZGlzdHJpYnV0aW9uIGlzIGF0IHN1YmplY3QgbGV2ZWwgd2UgY2FuIGNoZWNrIHdoZXRoZXIgdGhlIG9yZGVyIG9mIHN1YmplY3RzIGluIHRoaXMgZGlzdHJpYnV0aW9uIGRlcGVuZHMgb24gdGhlaXIgb3JkZXIgaW4gdGhlIGRpc3RyaWJ1dGlvbiBvZiBkYXlzIGJldHdlZW4gY29tcGxldGluZyB0aGUgdHdvIHRlc3RzLgoKTWFrZSBkYXRhIGZyYW1lIHdpdGggZGlmZmVyZW5jZSBiZXR3ZWVuIHR3byBzY29yZXMgZm9yIGVhY2ggbWVhc3VyZSBmb3IgZWFjaCBzdWJqZWN0LiBTaW5jZSB0aGUgc2NvcmVzIGZvciBkaWZmZXJlbnQgbWVhc3VyZXMgYXJlIG9uIGRpZmZlcmVudCBzY2FsZXMgZm9yIGNvbXBhcmFiaWxpdHkgdGhlIGRpZmZlcmVuY2Ugc2NvcmVzIGFyZSBub3JtYWxpemVkIChpZSBkZW1lYW5lZCBhbmQgZGl2aWRpZWQgYnkgdGhlIHNkIG9mIHRoZSBkaWZmZXJlbmNlIHNjb3JlIGRpc3RyaWJ1dGlvbi4pIE5vdGUgdGhhdCB0aGUgdmFyaWFuY2Ugb2YgdGhlIGRpZmZlcmVuY2Ugc2NvcmUgZGlzdHJpYnV0aW9uIGFjY291bnRzIGZvciB0aGUgdmFyaWFuY2UgaW4gYm90aCB0aW1lIHBvaW50cyBieSBzdW1taW5nIHRoZW0uIE5vcm1hbGl6YXRpb24gZXF1YXRlcyB0aGUgbWVhbnMgb2YgZWFjaCBkaWZmZXJlbmNlIHNjb3JlIGRpc3RyaWJ1dGlvbiB0byAwIHdoaWNoIHdvdWxkIG1hc2sgYW55IG1lYW5pbmdmdWwgY2hhbmdlIGJldHdlZW4gdGhlIHR3byB0aW1lIHBvaW50cyBidXQgdGhlIGFuYWx5c2lzIGhlcmUgZG9lcyBub3QgaW50ZXJwcmV0IHRoZSBtZWFuIG9mIHRoZSBkaWZmZXJlbmNlIHNjb3JlIGRpc3RyaWJ1dGlvbnMgYnV0IGlzIGludGVyZXN0ZWQgaW4gaXRzIHJlbGF0aW9uIHRvIHRoZSBkYXlzIGJldHdlZW4gY29tcGxldGlvbi4gV2UgY2hlY2sgaWYgdGhlIHZhcmlhYmxlcyBzaG93IHN5c3RlbWF0aWMgZGlmZmVyZW5jZXMgYmV0d2VlbiB0aGUgdHdvIHBvaW50cyBsYXRlci4KCkhlcmUgd2UgY2hlY2sgaWYgdGhlIGRpZmZlcmVuY2UgaXMgbGFyZ2VyIHRoZSBsb25nZXIgdGhlIGRlbGF5CgpgYGB7ciB3YXJuaW5nPUZBTFNFfQpudW1lcmljX2NvbHMgPSBjKCkKCmZvcihpIGluIDE6bGVuZ3RoKG5hbWVzKHRlc3RfZGF0YSkpKXsKICBpZihpcy5udW1lcmljKHRlc3RfZGF0YVssaV0pICYgbmFtZXModGVzdF9kYXRhKVtpXSAlaW4lIG5hbWVzKHJldGVzdF9kYXRhKSl7CiAgICBudW1lcmljX2NvbHMgPC0gYyhudW1lcmljX2NvbHMsIG5hbWVzKHRlc3RfZGF0YSlbaV0pCiAgfQp9Cgp0MV8yX2RpZmZlcmVuY2UgPSBkYXRhLmZyYW1lKCkKCmZvcihpIGluIDE6bGVuZ3RoKG51bWVyaWNfY29scykpewogIHRtcCA9IG1hdGNoX3QxX3QyKG51bWVyaWNfY29sc1tpXSxmb3JtYXQ9J3dpZGUnKQogIHRtcCA9IHRtcCAlPiUgCiAgbXV0YXRlKGRpZmZlcmVuY2UgPSBzY2FsZShgMmAgLSBgMWApKQogIHQxXzJfZGlmZmVyZW5jZSA9IHJiaW5kKHQxXzJfZGlmZmVyZW5jZSwgdG1wKQp9Cgp0MV8yX2RpZmZlcmVuY2UkZGlmZmVyZW5jZSA9IGFzLmRhdGEuZnJhbWUodDFfMl9kaWZmZXJlbmNlJGRpZmZlcmVuY2UpJFYxCgp0MV8yX2RpZmZlcmVuY2UgPSB0MV8yX2RpZmZlcmVuY2UgJT4lIHNlcGFyYXRlKGR2LCBjKCJ0YXNrIiwgImR2MiIpLCBzZXA9IlxcLiIsIHJlbW92ZT1GQUxTRSkKYGBgCgpgYGB7ciBlY2hvPUZBTFNFfQpybSh0bXAsIGkpCmBgYAoKQWRkIGNvbXBsZXRpb24gZGF0ZXMgdG8gdGhpcyBkYXRhIGZyYW1lLgoKYGBge3J9CnJldGVzdF90YXNrX2NvbXBfdGltZXMgPSByZWFkLmNzdihwYXN0ZTAocmV0ZXN0X2RhdGFfcGF0aCwgJ0xvY2FsL3JldGVzdF90YXNrX2NvbXBsZXRpb25fdGltZXMuY3N2JykpCnRlc3RfdGFza19jb21wX3RpbWVzID0gcmVhZC5jc3YocGFzdGUwKHJldGVzdF9kYXRhX3BhdGgsICdMb2NhbC90ZXN0X3Rhc2tfY29tcGxldGlvbl90aW1lcy5jc3YnKSkKdGFza19jb21wX3RpbWVzID0gbWVyZ2UocmV0ZXN0X3Rhc2tfY29tcF90aW1lcywgdGVzdF90YXNrX2NvbXBfdGltZXMsIGJ5PWMoJ3dvcmtlcl9pZCcsJ3Rhc2snKSkKcm0ocmV0ZXN0X3Rhc2tfY29tcF90aW1lcywgdGVzdF90YXNrX2NvbXBfdGltZXMpCnRhc2tfY29tcF90aW1lcyA9IHRhc2tfY29tcF90aW1lcyAlPiUKICBzZWxlY3QoLVgueCwgLVgueSkgJT4lCiAgbXV0YXRlKGZpbmlzaF9kYXkueCA9IGFzLkRhdGUoZmluaXNoX2RheS54KSwKICAgICAgICAgZmluaXNoX2RheS55ID0gYXMuRGF0ZShmaW5pc2hfZGF5LnkpLAogICAgICAgICBkYXlzX2J0dyA9IGZpbmlzaF9kYXkueC1maW5pc2hfZGF5LnkpICU+JQogIHJlbmFtZShzdWJfaWQ9d29ya2VyX2lkKQpgYGAKCmBgYHtyIHdhcm5pbmc9RkFMU0V9CnQxXzJfZGlmZmVyZW5jZSA9IG1lcmdlKHQxXzJfZGlmZmVyZW5jZSwgdGFza19jb21wX3RpbWVzWyxjKCdzdWJfaWQnLCAndGFzaycsJ2RheXNfYnR3JyldLCBieT1jKCdzdWJfaWQnLCAndGFzaycpKQpgYGAKCmBgYHtyIGVjaG89RkFMU0V9CnJtKHRhc2tfY29tcF90aW1lcykKYGBgCgpXaGF0IGRvZXMgdGhlIGRpc3RyaWJ1dGlvbiBvZiBkaWZmZXJlbmNlcyBsb29rIGxpa2U6IFRoZSBkaXN0cmlidXRpb24gb2YgZGlmZmVyZW5jZXMgYmV0d2VlbiB0d28gdGltZSBwb2ludHMgZm9yIGVhY2ggbWVhc3VyZSAKCmBgYHtyIHdhcm5pbmc9RkFMU0UsIG1lc3NhZ2U9RkFMU0V9CnQxXzJfZGlmZmVyZW5jZSAlPiUKICBnZ3Bsb3QoYWVzKGRpZmZlcmVuY2UsIGFscGhhPWR2KSkrCiAgZ2VvbV9oaXN0b2dyYW0ocG9zaXRpb249J2lkZW50aXR5JykrCiAgdGhlbWUobGVnZW5kLnBvc2l0aW9uID0gJ25vbmUnKQpgYGAKCkhvdyBkbyB0aGUgZGlmZmVyZW5jZSBzY29yZSBkaXN0cmlidXRpb25zIGxvb2sgbGlrZSB3aXRoIHJlc3BlY3QgdG8gdGhlIGRheXMgYmV0d2VlbiBjb21wbGV0aW9uPwoKYGBge3J9CnQxXzJfZGlmZmVyZW5jZSAlPiUKICBnZ3Bsb3QoKSsKICBnZW9tX3Ntb290aChhZXMoYXMubnVtZXJpYyhkYXlzX2J0dyksIGFicyhkaWZmZXJlbmNlKSwgZ3JvdXA9ZmFjdG9yKGR2KSksIG1ldGhvZD0nbG0nLCBzZT1GQUxTRSkrCiAgZ2VvbV9zbW9vdGgoYWVzKGFzLm51bWVyaWMoZGF5c19idHcpLCBhYnMoZGlmZmVyZW5jZSkpLCBtZXRob2Q9J2xtJywgY29sb3IgPSAiYmxhY2siLCBzZT1GQUxTRSkrCiAgdGhlbWUobGVnZW5kLnRpdGxlID0gZWxlbWVudF9ibGFuaygpKSsKICB4bGFiKCdEYXlzIGJldHdlZW4gY29tcGxldGlvbicpKwogIHlsYWIoJ0Fic29sdXRlIFNjYWxlZCBkaWZmZXJlbmNlIHNjb3JlJykKCmBgYAoKVG8gdGVzdCBpZiB0aGUgc2xvcGUgb2YgdGhlIGJsYWNrIGlzIHNpZ25pZmljYW50IHdlIHdvdWxkIHJ1biBhIG1peGVkIGVmZmVjdHMgbW9kZWwgd2l0aCBhIGZpeGVkIGVmZmVjdCBmb3IgZGF5cyBiZXR3ZWVuIGNvbXBsZXRpb24sIHJhbmRvbSBzbG9wZSBmb3IgZWFjaCBkdiBkZXBlbmRpbmcgb24gdGhlIGRheXMgYmV0d2VlbiBhbmQgcmFuZG9tIGludGVyY2VwdCBmb3IgZWFjaCBkdi4KCkJlZm9yZSBJIHdhcyB1c2luZyBzdWJqZWN0cyBhcyBhIHJhbmRvbSBlZmZlY3QgYnV0IGRheXMgYmV0d2VlbiB0aGUgdHdvIHRpbWUgcG9pbnRzIGZvciBlYWNoIG1lYXN1cmUgZGVwZW5kcyBvbiBzdWJqIGlkLiBXaGF0IHZhcmllcyByYW5kb21seSBpcyB3aGljaCBkdiB3ZSBhcmUgbG9va2luZyBmb3IgaXRzIGRpc3RyaWJ1dGlvbiBvZiBkaWZmZXJlbmNlcyBpbiByZWxhdGlvbiB0byB0aGUgZGF5cyBiZXR3ZWVuIHRoZSB0aW1lIHBvaW50cy4gU28gSSBjaGFuZ2VkIHRoZSBtb2RlbCB0byBoYXZlIGZpeGVkIGVmZmVjdCBmb3IgdGhlIGRheXMgYmV0d2VlbiwgYSByYW5kb20gc2xvcGUgZm9yIChkZXBlbmRlbnQgdmFyaWFibGVzIGNhbiBiZSBkaWZmZXJlbnRpYWxseSBzZW5zaXRpdmUgdG8gdGggZWZmZWN0IG9mIGRheXMgYmV0d2VlbikgYW5kIGEgcmFuZG9tIGludGVyY2VwdCBmb3IgZGVwZW5kZW50IHZhcmlhYmxlLgoKU2lnbmlmaWNhbnQgZml4ZWQgZWZmZWN0IHN1Z2dlc3RzIHRoYXQgb24gYXZlcmFnZSB0aGUgbG9uZ2VyIHRoZSBkZWxheSB0aGUgc21hbGxlciB0aGUgZGlmZmVyZW5jZS4KCmBgYHtyfQpzdW1tYXJ5KGxtZXJUZXN0OjpsbWVyKGFicyhkaWZmZXJlbmNlKSB+IHNjYWxlKGRheXNfYnR3KSsoc2NhbGUoZGF5c19idHcpIHwgZHYpLCBkYXRhPXQxXzJfZGlmZmVyZW5jZSkpIApgYGAKCkJ1dCBpZiBJIGp1c3QgcnVuIG9uZSBtaXhlZCBlZmZlY3RzIG1vZGVsIHRoZW4gd2UgZG9uJ3QgZ2V0IGEgc2Vuc2Ugb2YgdGhlIHNpbXBsZSBlZmZlY3RzIChob3cgbWFueSBvZiB0aGUgdmFyaWFibGVzIHRoaXMgZWZmZWN0IG9mIHRoZSBkYXlzIGJldHdlZW4gaXMgc2lnbmlmaWNhbnQgYW5kIGluIHdoaWNoIGRpcmVjdGlvbikuIEkgY2FuIHJ1biBpdCBzZXBhcmF0ZWx5IGZvciBlYWNoIGR2IHRvIHNlZSBpZiBhbGwgZGlmZmVyZW5jZSBzY29yZSBkaXN0cmlidXRpb25zIGFyZSBhZmZlY3RlZCB0aGUgc2FtZSB3YXkgZGVwZW5kaW5nIG9uIHRoZSBkYXlzIGJldHdlZW4gY29tcGxldGlvbi4KCmBgYHtyfQpnZXRfZGVsYXlfZWZmZWN0ID0gZnVuY3Rpb24oZGYpewogIG1vZCA9IGxtKGFicyhkaWZmZXJlbmNlKSB+IHNjYWxlKGRheXNfYnR3KSwgZGF0YSA9IGRmKQogIG91dCA9IGRhdGEuZnJhbWUoZXN0aW1hdGU9Y29lZihzdW1tYXJ5KG1vZCkpWyJzY2FsZShkYXlzX2J0dykiLCJFc3RpbWF0ZSJdLCBwdmFsPWNvZWYoc3VtbWFyeShtb2QpKVsic2NhbGUoZGF5c19idHcpIiwiUHIoPnx0fCkiXSkKICByZXR1cm4ob3V0KQp9CgpzaWdfZGF5c19lZmZlY3QgPSB0MV8yX2RpZmZlcmVuY2UgJT4lCiAgZ3JvdXBfYnkoZHYpICU+JQogIGRvKGdldF9kZWxheV9lZmZlY3QoLikpICU+JQogIGZpbHRlcihwdmFsPDAuMDUpCgpzaWdfZGF5c19lZmZlY3QKYGBgCgpGb3IgdmlzdWFsaXphdGlvbiBJIHVzZWQgdG8gc3VtbWFyaXplIHRoZSBkaWZmZXJlbmNlIHNjb3JlcyBwZXIgcGVyc29uIGJ5IGxvb2tpbmcgYXQgdGhlIGF2ZXJhZ2UgZGlmZmVyZW5jZSBwZXIgdGFzayBwZXIgc3ViamVjdCBhbmQgcGxvdCB0aGF0IGFnYWluc3QgdGhlIG51bWJlciBvZiBkYXlzIGJldHdlZW4gY29tcGxldGlvbi4gVGhpcyB5aWVsZHMgbXVsdGlwbGUgcG9pbnRzIGZvciBlYWNoIHZhbHVlIG9uIHggcmVwcmVzZW50aW5nIHRoZSBhdmVyYWdlIGRpZmZlcmVuY2UgcGVyIHRhc2sgZm9yIGVhY2ggc3ViamVjdC4gQnV0IHZhcmlhYmxlcyB3aXRoaW4gYSB0YXNrIGNvdWxkIGdvIGluIGRpZmZlcmVudCBkaXJlY3Rpb25zIChlLmcuIGlmIHBhdGllbnQgcHJvcG9ydGlvbiBpbmNyZWFzZXMgZGlzY291bnQgcmF0ZSBkZWNyZWFzZXMpIHNvIHRoaXMgZG9lc24ndCBzZWVtIGxpa2UgYSBnb29kIGlkZWEpIAoKSW5zdGVhZCBJIG5vdyBwbG90IGFsbCB0aGUgZGF0YSBncm91cGluZyBieSBkZXBlbmRhbnQgdmFyaWFibGUgYW5kIGNvbG9yaW5nIGRlcGVuZGluZyBvbiB3aGV0aGVyIHRoZSBkaWZmZXJlbmNlIHNjb3JlIGRpc3RyaWJ1dGlvbiBmb3IgdGhhdCB2YXJpYWJsZSBoYXMgYSBzaWduaWZpY2FudCBzbG9wZSB3aGVuIHJlZ3Jlc3NlZCBvdmVyIGRheXMgYmV0d2Vlbi4gVGhlIGJsYWNrIGlzIHRoZSBtYWluIGVmZmVjdCBmb3IgdGhlIGxhcmdlIG11bHRpbGV2ZWwgbW9kZWwgKHRoaXMgaXMgdGhlIHNhbWUgcGxvdCBhcyBhYm92ZSBjb2xvcmVkIGJ5IHdoZXRoZXIgdGhlIGRpZmZlcmVuY2Ugc2NvcmUgZGlzdHJpYnV0aW9uIGZvciBlYWNoIGR2IGhhcyBhIHNpZ25pZmljYW50IHNsb3BlIGRlcGVuZGluZyBvbiBkYXlzIGJldHdlZW4pLgoKYGBge3J9CnQxXzJfZGlmZmVyZW5jZSAlPiUKICBtdXRhdGUoc2lnX2RheXNfZWZmZWN0ID0gaWZlbHNlKGR2ICVpbiUgc2lnX2RheXNfZWZmZWN0JGR2LCAxLCAwKSklPiUKICBhcnJhbmdlKC1zaWdfZGF5c19lZmZlY3QpICU+JQogIGdncGxvdCgpKwogIHN0YXRfc21vb3RoIChhZXMoYXMubnVtZXJpYyhkYXlzX2J0dyksIGFicyhkaWZmZXJlbmNlKSwgCiAgICAgICAgICAgICAgICAgIGdyb3VwPWZhY3RvcihkdiwgbGV2ZWxzPXVuaXF1ZShkdltvcmRlcihzaWdfZGF5c19lZmZlY3QpXSksIG9yZGVyZWQ9VFJVRSksIAogICAgICAgICAgICAgICAgICBjb2xvcj1mYWN0b3Ioc2lnX2RheXNfZWZmZWN0LCBsZXZlbHMgPSBjKDEsMCkpKSxnZW9tPSJsaW5lIiwgYWxwaGE9MC41LCBtZXRob2Q9J2xtJykrCiAgZ2VvbV9zbW9vdGgoYWVzKGFzLm51bWVyaWMoZGF5c19idHcpLCBhYnMoZGlmZmVyZW5jZSkpLCBtZXRob2Q9J2xtJywgY29sb3IgPSAiYmxhY2siLCBzZT1GQUxTRSkrCiAgdGhlbWUobGVnZW5kLnRpdGxlID0gZWxlbWVudF9ibGFuaygpKSsKICB4bGFiKCdEYXlzIGJldHdlZW4gY29tcGxldGlvbicpKwogIHlsYWIoJ1NjYWxlZCBkaWZmZXJlbmNlIHNjb3JlJykrCiAgc2NhbGVfY29sb3JfZGlzY3JldGUoYnJlYWtzPWMoMCwxKSwKICAgICAgICAgICAgICAgICAgICAgICBsYWJlbHM9YygiTlMiLCAiU2lnIikpCgpnZ3NhdmUoJ0RheXNCdHdFZmZlY3QuanBnJywgZGV2aWNlID0gImpwZWciLCBwYXRoID0gIi9Vc2Vycy96ZXluZXBlbmthdmkvRHJvcGJveC9Qb2xkcmFja0xhYi9TUk9fUmV0ZXN0X0FuYWx5c2VzL291dHB1dC9maWd1cmVzLyIsIHdpZHRoID0gNiwgaGVpZ2h0ID0gNCwgdW5pdHMgPSAiaW4iLCBsaW1pdHNpemUgPSBGQUxTRSwgZHBpID0gMTAwKQpgYGAKCkNvbmNsdXNpb246IFRoZSBhdmVyYWdlIGRpZmZlcmVuY2UgYmV0d2VlbiB0aGUgc2NvcmVzIG9mIGEgbWVhc3VyZSBpZiBhbnl0aGluZyBkZWNyZWFzZXMgd2l0aCB0aGUgaW5jcmVhc2VkIGRlbGF5LiBUaGlzIGZpeGVkIGVmZmVjdCBtYXNrcyB0aGUgc2ltcGxlIGVmZmVjdCBmb3IgZWFjaCBkdi4gV2hlbiBsb29rZWQgYXQgaW5kaXZpZHVhbGx5IG9ubHkgMTAgdmFyaWFibGVzIHNob3cgYSBkZXBlbmRlbmNlIG9uIGRheXMgYmV0d2VlbiBjb21wbGV0aW9uLiBXaGlsZSA3IG9mIHRoZW0gc2hvdyBkZWNyZWFzaW5nIGRpZmZlcmVuY2VzIGRlcGVuZGluZyBvbiBkYXlzIGJldHdlZW4gMyBkbyBzaG93IGluY3JlYXNpbmcgZGlmZmVyZW5jZXMuClRoaXMgc3VnZ2VzdHMgdGhhdCB0aGUgbGFjayBvZiBzdHJpY3RlciBjb250cm9sIG92ZXIgdGhlIGRheXMgYmV0d2VlbiBjb21wbGV0aW9uIG9mIHRoZSBtZWFzdXJlbWVudCBkb2VzIG5vdCBoYXZlIGEgbGFyZ2UgZWZmZWN0IG9uIHRoZSBzdGFiaWxpdHkgb2YgbW9zdCBtZWFzdXJlcy4KCmVmZmVjdCBvZiB3aGV0aGVyIGRheXMgYmV0d2VlbiBsZWFkcyB0byBhIHNpZyBkaWZmZXJlbmNlIG9uIHJlbGlhYmlsaXR5IG9mIHRoYXQgdmFyaWFibGU/IHBsb3QgY29lZiBvZiBzaWdfZGF5X2VmZmVjdCBvdmVyIHJlbF9kZiBwb2ludCBlc3RpbWF0ZQoKRG8gdGhlIHZhcmlhYmxlcyB0aGF0IHNob3cgc2lnbmlmaWNhbnQgZGVwZW5kZW5jZSBvbiB0aGUgZGVsYXkgYmV0d2VlbiB0aGUgY29tcGxldGlvbiB0aW1lcyBoYXZlIHN5c3RlbWF0aWNhbGx5IGxvd2VyL2hpZ2hlciByZWxpYWJpbGl0eT8gTG9va2luZyBhdCB0aGUgcmVsaWFiaWxpdHkgcG9pbnQgZXN0aW1hdGUgb3ZlciB0aGUgc2l6ZSBvZiB0aGUgZGVsYXkgZWZmZWN0LiBUaGVyZSBhcmUgdmVyeSBmZXcgdmFyaWFibGVzIHRvIGNvbXBhcmUgYW55d2F5IGJ1dCByZWdhcmRsZXNzIGRvZXNuJ3QgbG9vayBjb25jbHVzaXZlL2NvbmNlcm5pbmcuCgpgYGB7cn0KI0NyZWF0ZSBkZiBvZiBwb2ludCBlc3RpbWF0ZSByZWxpYWJpbGl0aWVzCnJlbF9kZiA8LSBkYXRhLmZyYW1lKHNwZWFybWFuID0gcmVwKE5BLCBsZW5ndGgobnVtZXJpY19jb2xzKSksCiAgICAgICAgICAgICAgICAgICAgIGljYyA9IHJlcChOQSwgbGVuZ3RoKG51bWVyaWNfY29scykpLAogICAgICAgICAgICAgICAgICAgICBwZWFyc29uID0gcmVwKE5BLCBsZW5ndGgobnVtZXJpY19jb2xzKSksCiAgICAgICAgICAgICAgICAgICAgIHBhcnRpYWxfZXRhX3NxID0gcmVwKE5BLCBsZW5ndGgobnVtZXJpY19jb2xzKSksCiAgICAgICAgICAgICAgICAgICAgIHNlbSA9IHJlcChOQSwgbGVuZ3RoKG51bWVyaWNfY29scykpLAogICAgICAgICAgICAgICAgICAgICB2YXJfc3VicyA9IHJlcChOQSwgbGVuZ3RoKG51bWVyaWNfY29scykpLAogICAgICAgICAgICAgICAgICAgICB2YXJfaW5kID0gcmVwKE5BLCBsZW5ndGgobnVtZXJpY19jb2xzKSksCiAgICAgICAgICAgICAgICAgICAgIHZhcl9yZXNpZCA9IHJlcChOQSwgbGVuZ3RoKG51bWVyaWNfY29scykpKQoKcm93Lm5hbWVzKHJlbF9kZikgPC0gbnVtZXJpY19jb2xzCgpmb3IoaSBpbiAxOmxlbmd0aChudW1lcmljX2NvbHMpKXsKICByZWxfZGZbbnVtZXJpY19jb2xzW2ldLCAnc3BlYXJtYW4nXSA8LSBnZXRfc3BlYXJtYW4obnVtZXJpY19jb2xzW2ldKSAKICByZWxfZGZbbnVtZXJpY19jb2xzW2ldLCAnaWNjJ10gPC0gZ2V0X2ljYyhudW1lcmljX2NvbHNbaV0pCiAgcmVsX2RmW251bWVyaWNfY29sc1tpXSwgJ3BlYXJzb24nXSA8LSBnZXRfcGVhcnNvbihudW1lcmljX2NvbHNbaV0pCiAgcmVsX2RmW251bWVyaWNfY29sc1tpXSwgJ3BhcnRpYWxfZXRhX3NxJ10gPC0gZ2V0X3BhcnRpYWxfZXRhKG51bWVyaWNfY29sc1tpXSkKICByZWxfZGZbbnVtZXJpY19jb2xzW2ldLCAnc2VtJ10gPC0gZ2V0X3NlbShudW1lcmljX2NvbHNbaV0pCiAgcmVsX2RmW251bWVyaWNfY29sc1tpXSwgJ3Zhcl9zdWJzJ10gPC0gZ2V0X3Zhcl9icmVha2Rvd24obnVtZXJpY19jb2xzW2ldKSRzdWJzCiAgcmVsX2RmW251bWVyaWNfY29sc1tpXSwgJ3Zhcl9pbmQnXSA8LSBnZXRfdmFyX2JyZWFrZG93bihudW1lcmljX2NvbHNbaV0pJGluZAogIHJlbF9kZltudW1lcmljX2NvbHNbaV0sICd2YXJfcmVzaWQnXSA8LSBnZXRfdmFyX2JyZWFrZG93bihudW1lcmljX2NvbHNbaV0pJHJlc2lkCn0KCnJlbF9kZiRkdiA9IHJvdy5uYW1lcyhyZWxfZGYpCnJvdy5uYW1lcyhyZWxfZGYpID0gc2VxKDE6bnJvdyhyZWxfZGYpKQpyZWxfZGYkdGFzayA9ICd0YXNrJwpyZWxfZGZbZ3JlcCgnc3VydmV5JywgcmVsX2RmJGR2KSwgJ3Rhc2snXSA9ICdzdXJ2ZXknCnJlbF9kZltncmVwKCdob2x0JywgcmVsX2RmJGR2KSwgJ3Rhc2snXSA9ICJ0YXNrIgpyZWxfZGYgPSByZWxfZGYgJT4lCiAgc2VsZWN0KGR2LCB0YXNrLCBzcGVhcm1hbiwgaWNjLCBwZWFyc29uLCBwYXJ0aWFsX2V0YV9zcSwgc2VtLCB2YXJfc3VicywgdmFyX2luZCwgdmFyX3Jlc2lkKQojIHJvdy5uYW1lcyhyZWxfZGYpID0gTlVMTApgYGAKCmBgYHtyfQpzaWdfZGF5c19lZmZlY3QgJT4lCiAgbGVmdF9qb2luKHJlbF9kZiwgYnk9J2R2JykgJT4lCiAgZ2dwbG90KGFlcyhlc3RpbWF0ZSwgaWNjKSkrCiAgZ2VvbV9wb2ludCgpCmBgYAoKIyMjIE92ZXJsYXBwaW5nIHN1cnZlcnkgcXVlc3Rpb25zCgpTb21lIHN1cnZleXMgaGF2ZSBvdmVybGFwcGluZyBxdWVzdGlvbnMuIERvIHRoZXNlIGNvcnJlbGF0ZSB3aXRoaW4gYW5kIGFjcm9zcyBzZXNzaW9ucz8KCkZpcnN0IGRldGVybWluZSB0aGUgb3ZlcmxhcHBpbmcgcXVlc3Rpb25zLgoKYGBge3J9CnRtcCA9IHJlYWQuY3N2KGd6ZmlsZShwYXN0ZTAocmV0ZXN0X2RhdGFfcGF0aCwgJ2l0ZW1zLmNzdi5neicpKSkKdG1wID0gdG1wICU+JSAKICBmaWx0ZXIod29ya2VyID09ICdzMDA1JykgJT4lIAogIHNlbGVjdChpdGVtX0lELCBpdGVtX3RleHQpICU+JSAKICBtdXRhdGUoaXRlbV90ZXh0ID0gdHJpbXdzKGFzLmNoYXJhY3RlcihpdGVtX3RleHQpKSkgJT4lCiAgdW5pdGUoaXRlbSwgYygiaXRlbV9JRCIsICJpdGVtX3RleHQiKSwgc2VwID0gIl9fXyIpCgpjb21iID0gYXMuZGF0YS5mcmFtZSh0KGNvbWJuKHVuaXF1ZSh0bXAkaXRlbSksMikpKQoKZHVwbGljYXRlX2l0ZW1zID0gY29tYiAlPiUgCiAgZmlsdGVyKGdyZXBsKCdkb3NwZXJ0JywgVjEpPT1GQUxTRSkgJT4lCiAgZmlsdGVyKGdyZXBsKCdzZWxlY3Rpb25fb3B0aW1pemF0aW9uJywgVjEpPT1GQUxTRSkgJT4lCiAgZmlsdGVyKGdyZXBsKCdzZW5zYXRpb25fc2Vla2luZycsIFYxKT09RkFMU0UpICU+JQogIHNlcGFyYXRlKFYxLCBjKCJpdGVtMV9JRCIsICJpdGVtMV90ZXh0IiksIHNlcD0iX19fIikgJT4lCiAgc2VwYXJhdGUoVjIsIGMoIml0ZW0yX0lEIiwgIml0ZW0yX3RleHQiKSwgc2VwPSJfX18iKSAlPiUKICBtdXRhdGUoc2ltaWxhcml0eSA9IGxldmVuc2h0ZWluU2ltKGl0ZW0xX3RleHQsIGl0ZW0yX3RleHQpKSAlPiUKICBmaWx0ZXIoc2ltaWxhcml0eT4wLjgpICU+JQogIHNlbGVjdChpdGVtMV9JRCwgaXRlbTJfSUQsIGl0ZW0xX3RleHQsIGl0ZW0yX3RleHQpCgpkdXBsaWNhdGVfaXRlbXMKYGBgCgpgYGB7ciB3YXJuaW5nPUZBTFNFLCBtZXNzYWdlPUZBTFNFfQojc3VydmV5cyB0byByZWFkIGluCmV4dHJhY3RfaXRlbXMgPSBjKCd3b3JrZXInLHVuaXF1ZSh3aXRoKGR1cGxpY2F0ZV9pdGVtcywgYyhpdGVtMV9JRCwgaXRlbTJfSUQpKSkpCgojY29ycmVsYXRpb25zIHRvIGNvbXB1dGU6CiNpdGVtMV90MSAtIGl0ZW0yX3QxLCAKI2l0ZW0xX3QyIC0gaXRlbTJfdDIsIAojaXRlbTFfdDEgLSBpdGVtMl90MiwgCiNpdGVtMV90MiAtIGl0ZW0yX3QxCgpkdXBsaWNhdGVfaXRlbXNfZGF0YV90MSA9IHJlYWQuY3N2KHBhc3RlMCh0ZXN0X2RhdGFfcGF0aCwgJ3N1YmplY3RfeF9pdGVtcy5jc3YnKSkKZHVwbGljYXRlX2l0ZW1zX2RhdGFfdDIgPSByZWFkLmNzdihwYXN0ZTAocmV0ZXN0X2RhdGFfcGF0aCwgJ3N1YmplY3RfeF9pdGVtcy5jc3YnKSkKCmR1cGxpY2F0ZV9pdGVtc19kYXRhX3QxID0gZHVwbGljYXRlX2l0ZW1zX2RhdGFfdDEgJT4lCiAgZmlsdGVyKHdvcmtlciAlaW4lIGR1cGxpY2F0ZV9pdGVtc19kYXRhX3QyJHdvcmtlcikgJT4lCiAgc2VsZWN0KGV4dHJhY3RfaXRlbXMpCgpkdXBsaWNhdGVfaXRlbXNfZGF0YV90Mj1kdXBsaWNhdGVfaXRlbXNfZGF0YV90MiAlPiUKICBmaWx0ZXIod29ya2VyICVpbiUgZHVwbGljYXRlX2l0ZW1zX2RhdGFfdDEkd29ya2VyKSAlPiUKICBzZWxlY3QoZXh0cmFjdF9pdGVtcykKCmR1cGxpY2F0ZV9pdGVtcyA9IGR1cGxpY2F0ZV9pdGVtcyAlPiUKICBtdXRhdGUodDFfdDFfY29yID0gTkEsCiAgICAgICAgIHQyX3QyX2NvciA9IE5BLAogICAgICAgICB0MV90Ml9jb3IgPSBOQSwKICAgICAgICAgdDJfdDFfY29yID0gTkEsCiAgICAgICAgIHQxX3QxX3BvbHljb3IgPSBOQSwKICAgICAgICAgdDJfdDJfcG9seWNvciA9IE5BLAogICAgICAgICB0MV90Ml9wb2x5Y29yID0gTkEsCiAgICAgICAgIHQyX3QxX3BvbHljb3IgPSBOQSkKCmZvcihpIGluIDE6bnJvdyhkdXBsaWNhdGVfaXRlbXMpKXsKICBkdXBsaWNhdGVfaXRlbXMkdDFfdDFfY29yW2ldID0gYWJzKGNvcihkdXBsaWNhdGVfaXRlbXNfZGF0YV90MVssYyhkdXBsaWNhdGVfaXRlbXMkaXRlbTFfSURbaV0pXSwKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGR1cGxpY2F0ZV9pdGVtc19kYXRhX3QxWyxjKGR1cGxpY2F0ZV9pdGVtcyRpdGVtMl9JRFtpXSldKSkKICAKICBkdXBsaWNhdGVfaXRlbXMkdDJfdDJfY29yW2ldID0gYWJzKGNvcihkdXBsaWNhdGVfaXRlbXNfZGF0YV90MlssYyhkdXBsaWNhdGVfaXRlbXMkaXRlbTFfSURbaV0pXSwKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGR1cGxpY2F0ZV9pdGVtc19kYXRhX3QyWyxjKGR1cGxpY2F0ZV9pdGVtcyRpdGVtMl9JRFtpXSldKSkKICAKICBkdXBsaWNhdGVfaXRlbXMkdDFfdDJfY29yW2ldID0gYWJzKGNvcihkdXBsaWNhdGVfaXRlbXNfZGF0YV90MVssYyhkdXBsaWNhdGVfaXRlbXMkaXRlbTFfSURbaV0pXSwKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGR1cGxpY2F0ZV9pdGVtc19kYXRhX3QyWyxjKGR1cGxpY2F0ZV9pdGVtcyRpdGVtMl9JRFtpXSldKSkKICAKICBkdXBsaWNhdGVfaXRlbXMkdDJfdDFfY29yW2ldID0gYWJzKGNvcihkdXBsaWNhdGVfaXRlbXNfZGF0YV90MlssYyhkdXBsaWNhdGVfaXRlbXMkaXRlbTFfSURbaV0pXSwKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGR1cGxpY2F0ZV9pdGVtc19kYXRhX3QxWyxjKGR1cGxpY2F0ZV9pdGVtcyRpdGVtMl9JRFtpXSldKSkKICAKICBkdXBsaWNhdGVfaXRlbXMkdDFfdDFfcG9seWNvcltpXSA9IGFicyhwb2x5Y2hvcmljKGRhdGEuZnJhbWUoZHVwbGljYXRlX2l0ZW1zX2RhdGFfdDFbLGMoZHVwbGljYXRlX2l0ZW1zJGl0ZW0xX0lEW2ldKV0sCiAgICAgICAgICAgICAgICAgICAgICBkdXBsaWNhdGVfaXRlbXNfZGF0YV90MVssYyhkdXBsaWNhdGVfaXRlbXMkaXRlbTJfSURbaV0pXSkpJHJob1syXSkKCiAgZHVwbGljYXRlX2l0ZW1zJHQyX3QyX3BvbHljb3JbaV0gPSBhYnMocG9seWNob3JpYyhkYXRhLmZyYW1lKGR1cGxpY2F0ZV9pdGVtc19kYXRhX3QyWyxjKGR1cGxpY2F0ZV9pdGVtcyRpdGVtMV9JRFtpXSldLAogICAgICAgICAgICAgICAgICAgICAgZHVwbGljYXRlX2l0ZW1zX2RhdGFfdDJbLGMoZHVwbGljYXRlX2l0ZW1zJGl0ZW0yX0lEW2ldKV0pKSRyaG9bMl0pCgogIGR1cGxpY2F0ZV9pdGVtcyR0MV90Ml9wb2x5Y29yW2ldID0gYWJzKHBvbHljaG9yaWMoZGF0YS5mcmFtZShkdXBsaWNhdGVfaXRlbXNfZGF0YV90MVssYyhkdXBsaWNhdGVfaXRlbXMkaXRlbTFfSURbaV0pXSwKICAgICAgICAgICAgICAgICAgICAgIGR1cGxpY2F0ZV9pdGVtc19kYXRhX3QyWyxjKGR1cGxpY2F0ZV9pdGVtcyRpdGVtMl9JRFtpXSldKSkkcmhvWzJdKQoKICBkdXBsaWNhdGVfaXRlbXMkdDJfdDFfcG9seWNvcltpXSA9IGFicyhwb2x5Y2hvcmljKGRhdGEuZnJhbWUoZHVwbGljYXRlX2l0ZW1zX2RhdGFfdDJbLGMoZHVwbGljYXRlX2l0ZW1zJGl0ZW0xX0lEW2ldKV0sCiAgICAgICAgICAgICAgICAgICAgICBkdXBsaWNhdGVfaXRlbXNfZGF0YV90MVssYyhkdXBsaWNhdGVfaXRlbXMkaXRlbTJfSURbaV0pXSkpJHJob1syXSkKfQoKZHVwbGljYXRlX2l0ZW1zCmBgYAoKYGBge3J9CnN1bW1hcnkoZHVwbGljYXRlX2l0ZW1zJHQxX3QxX3BvbHljb3IpCnN1bW1hcnkoZHVwbGljYXRlX2l0ZW1zJHQyX3QyX3BvbHljb3IpCnN1bW1hcnkoZHVwbGljYXRlX2l0ZW1zJHQxX3QyX3BvbHljb3IpCnN1bW1hcnkoZHVwbGljYXRlX2l0ZW1zJHQyX3QxX3BvbHljb3IpCmBgYAoKYGBge3IgZWNobz1GQUxTRX0Kcm0odG1wLCBjb21iLCBzaWdfZGF5c19lZmZlY3QsIHQxXzJfZGlmZmVyZW5jZSwgZHVwbGljYXRlX2l0ZW1zLCBkdXBsaWNhdGVfaXRlbXNfZGF0YV90MSwgZHVwbGljYXRlX2l0ZW1zX2RhdGFfdDIsIGV4dHJhY3RfaXRlbXMsIGdldF9kZWxheV9lZmZlY3QpCmBgYAoKIyMgQ29tcGFyaXNvbiB0byBwcmlvciBsaXRlcmF0dXJlCgpSZWFkIGluIGFuZCBwcm9jZXNzIGJvb3RzdHJhcHBlZCByZXN1bHRzLgoKYGBge3J9CnByb2Nlc3NfYm9vdF9kZiA9IGZ1bmN0aW9uKGRmKXsKICBkZiA9IGRmICU+JQogIGRyb3BfbmEoKSAlPiUKICBtdXRhdGUoZHYgPSBhcy5jaGFyYWN0ZXIoZHYpLAogICAgICAgICBpY2MgPSBhcy5udW1lcmljKGFzLmNoYXJhY3RlcihpY2MpKSwKICAgICAgICAgc3BlYXJtYW4gPSBhcy5udW1lcmljKGFzLmNoYXJhY3RlcihzcGVhcm1hbikpLAogICAgICAgICBwZWFyc29uID0gYXMubnVtZXJpYyhhcy5jaGFyYWN0ZXIocGVhcnNvbikpLAogICAgICAgICBldGFfc3EgPSBhcy5udW1lcmljKGFzLmNoYXJhY3RlcihldGFfc3EpKSwKICAgICAgICAgc2VtID0gYXMubnVtZXJpYyhhcy5jaGFyYWN0ZXIoc2VtKSksCiAgICAgICAgIHBhcnRpYWxfZXRhX3NxID0gYXMubnVtZXJpYyhhcy5jaGFyYWN0ZXIocGFydGlhbF9ldGFfc3EpKSwKICAgICAgICAgb21lZ2Ffc3EgPSBhcy5udW1lcmljKGFzLmNoYXJhY3RlcihvbWVnYV9zcSkpLAogICAgICAgICB2YXJfc3VicyA9IGFzLm51bWVyaWMoYXMuY2hhcmFjdGVyKHZhcl9zdWJzKSksCiAgICAgICAgIHZhcl9pbmQgPSBhcy5udW1lcmljKGFzLmNoYXJhY3Rlcih2YXJfaW5kKSksCiAgICAgICAgIHZhcl9yZXNpZCA9IGFzLm51bWVyaWMoYXMuY2hhcmFjdGVyKHZhcl9yZXNpZCkpLAogICAgICAgICBGX3RpbWUgPSBhcy5udW1lcmljKGFzLmNoYXJhY3RlcihGX3RpbWUpKSwKICAgICAgICAgcF90aW1lID0gYXMubnVtZXJpYyhhcy5jaGFyYWN0ZXIocF90aW1lKSksCiAgICAgICAgIGRmX3RpbWUgPSBhcy5udW1lcmljKGFzLmNoYXJhY3RlcihkZl90aW1lKSksCiAgICAgICAgIGRmX3Jlc2lkID0gYXMubnVtZXJpYyhhcy5jaGFyYWN0ZXIoZGZfcmVzaWQpKSkKICByZXR1cm4oZGYpfSAKCmJvb3RfZGYgPC0gcmVhZC5jc3YoZ3pmaWxlKHBhc3RlMChyZXRlc3RfZGF0YV9wYXRoLCdib290c3RyYXBfbWVyZ2VkLmNzdi5neicpKSkKCmJvb3RfZGYgPSBwcm9jZXNzX2Jvb3RfZGYoYm9vdF9kZikKCmJvb3RfZGYgPSBib290X2RmW2Jvb3RfZGYkZHYgJWluJSByZXRlc3RfcmVwb3J0X3ZhcnMsXQoKIyBDaGVjayBpZiB5b3UgaGF2ZSBhbGwgdmFyaWFibGVzIGJvb3RzdHJhcHBlZAojIHJldGVzdF9yZXBvcnRfdmFyc1t3aGljaChyZXRlc3RfcmVwb3J0X3ZhcnMgJWluJSBib290X2RmJGR2PT1GQUxTRSldCgojIEJvb3QgZGYgY29udGFpbnMgaGRkbSBwYXJhbWV0ZXJzIGZpdCBvbiB0aGUgZnVsbCBzYW1wbGUgaW4gdGhlIHQxIGRhdGEKIyByZWZpdHNfYm9vdHN0cmFwX21lcmdlZC5jc3YuZ3ogY29udGFpbnMgYm9vdHN0cmFwcGVkIHJlbGlhYmlsaXRpZXMgCgpyZWZpdF9ib290X2RmID0gcmVhZC5jc3YoZ3pmaWxlKHBhc3RlMChyZXRlc3RfZGF0YV9wYXRoLCdyZWZpdHNfYm9vdHN0cmFwX21lcmdlZC5jc3YuZ3onKSkpCgpyZWZpdF9ib290X2RmID0gcHJvY2Vzc19ib290X2RmKHJlZml0X2Jvb3RfZGYpCgpmdWxsZml0X2Jvb3RfZGYgPSBib290X2RmW2FzLmNoYXJhY3Rlcihib290X2RmJGR2KSAlaW4lIHVuaXF1ZShhcy5jaGFyYWN0ZXIocmVmaXRfYm9vdF9kZiRkdikpLF0KCmJvb3RfZGYgPSBib290X2RmWyFhcy5jaGFyYWN0ZXIoYm9vdF9kZiRkdikgJWluJSB1bmlxdWUoYXMuY2hhcmFjdGVyKHJlZml0X2Jvb3RfZGYkZHYpKSxdCgpib290X2RmID0gcmJpbmQoYm9vdF9kZiwgcmVmaXRfYm9vdF9kZikKCnJtKHJlZml0X2Jvb3RfZGYpCmBgYAoKU3VtbWFyaXplIGJvb3RzdHJhcHBlZCByZXN1bHRzIGFuZCBtZXJnZSB0byBsaXQgcmV2aWV3IGRhdGEKCmBgYHtyIG1lc3NhZ2U9RkFMU0V9CnZhcl9ib290X2RmID0gYm9vdF9kZiAlPiUKICBncm91cF9ieShkdikgJT4lCiAgc3VtbWFyaXNlKG1lYW5faWNjID0gbWVhbihpY2MpLAogICAgICAgICAgICBtZWFuX3BlYXJzb24gPSBtZWFuKHBlYXJzb24pKQoKcmVsX2NvbXAgPSBsaXRfcmV2aWV3ICU+JQogIGxlZnRfam9pbih2YXJfYm9vdF9kZiwgYnkgPSAnZHYnKQpgYGAKCkhlcmUncyB3aGF0IG91ciBkYXRhIGxvb2tzIGxpa2U6ICg1ODMgZGF0YSBwb2ludHMgZm9yIDE3MSBtZWFzdXJlcykKCmBgYHtyfQpyZWxfY29tcApgYGAKCiMjIyBMaXQgcmV2aWV3IHJlc3VsdHMKCkRpc3RyaWJ1dGlvbiBvZiByZWxpYWJpbGl0aWVzLCBzYW1wbGUgc2l6ZXMgYW5kIGRlbGF5cwoKYGBge3J9CnJlbF9jb21wICU+JSAKICBzZWxlY3QoZHYsIHRhc2ssIHJldGVzdF9yZWxpYWJpbGl0eSwgc2FtcGxlX3NpemUsIGRheXMpICU+JQogIGZpbHRlcihkYXlzIDwgMzYwMCkgJT4lCiAgZ2F0aGVyKGtleSwgdmFsdWUsIC1kdiwgLXRhc2spICU+JQogIGdncGxvdChhZXModmFsdWUsIGZpbGw9dGFzaykpKwogIGdlb21fZGVuc2l0eShhbHBoYT0wLjUsIHBvc2l0aW9uPSdpZGVudGl0eScpKwogIGZhY2V0X3dyYXAofmtleSwgc2NhbGVzPSdmcmVlJykrCiAgdGhlbWUobGVnZW5kLnRpdGxlID0gZWxlbWVudF9ibGFuaygpKQpgYGAKCmBgYHtyfQpzdW1tYXJ5KHJlbF9jb21wJHNhbXBsZV9zaXplW3JlbF9jb21wJHRhc2sgPT0gInN1cnZleSJdKQpzdW1tYXJ5KHJlbF9jb21wJHNhbXBsZV9zaXplW3JlbF9jb21wJHRhc2sgPT0gInRhc2siXSkKYGBgCgpUaGUgbGl0ZXJhdHVyZSBoYXMgc21hbGxlciBzaXplZCBzYW1wbGVzIGZvciB0YXNrIG1lYXN1cmVzIGNvbXBhcmVkIHRvIHN1cnZleSBtZWFzdXJlcyB0aGF0IHJlcG9ydCByZXRlc3QgcmVsaWFiaWxpdHkuCgpgYGB7cn0Kc3VtbWFyeShsbShzYW1wbGVfc2l6ZSB+IHRhc2sscmVsX2NvbXApKQpgYGAKCldoYXQgcHJlZGljdHMgcmV0ZXN0IHJlbGlhYmlsaXR5IGluIHRoZSBsaXRlcmF0dXJlPwpUYXNrLCBzYW1wbGUgc2l6ZSwgZGF5cwoKYGBge3J9Cm1vZDEgPSBsbWVyKHJldGVzdF9yZWxpYWJpbGl0eSB+IHRhc2sgKyAoMXxkdiksIHJlbF9jb21wKQptb2QyID0gbG1lcihyZXRlc3RfcmVsaWFiaWxpdHkgfiB0YXNrICsgc2FtcGxlX3NpemUgKyAoMXxkdiksIHJlbF9jb21wKQoKYW5vdmEobW9kMSwgbW9kMikKYGBgCgpgYGB7cn0KbW9kMyA9IGxtZXIocmV0ZXN0X3JlbGlhYmlsaXR5IH4gdGFzayArIHNhbXBsZV9zaXplICsgZGF5cysgKDF8ZHYpLCByZWxfY29tcCkKCmFub3ZhKG1vZDIsIG1vZDMpCmBgYAoKYGBge3J9CnN1bW1hcnkobW9kMikKYGBgCgpUYXNrcyBoYXZlIHNpZ25pZmljYW50bHkgbG93ZXIgcmVsaWFiaWxpdHkgYW5kIHJlbGlhYmlsaXR5IGRlY3JlYXNlcyB3aXRoIGluY3JlYXNpbmcgc2FtcGxlIHNpemUuCgpgYGB7cn0KcmVsX2NvbXAgJT4lIAogIGdncGxvdChhZXMoc2FtcGxlX3NpemUsIHJldGVzdF9yZWxpYWJpbGl0eSwgY29sb3I9dGFzaykpKwogIGdlb21fc21vb3RoKG1ldGhvZD0nbG0nKSsKICBnZW9tX3BvaW50KGFscGhhID0gMC4yKSsKICB0aGVtZShsZWdlbmQudGl0bGUgPSBlbGVtZW50X2JsYW5rKCkpKwogIHhsYWIoIlNhbXBsZSBTaXplIikrCiAgeWxhYigiUmV0ZXN0IHJlbGlhYmlsaXR5IikKYGBgCgpJIHVzZWQgdG8gY29tcGFyZSBlZmZlY3Qgc2l6ZXMgb2YgZWZmZWN0IG9mIHRhc2sgb24gcmVsaWFiaWxpdHkgZXN0aW1hdGVzIGluIGxpdGVyYXR1cmUgdnMgb3VyIHJlc3VsdHMgKGp1c3QgbG9va2luZyBhdCB0aGUgdmFyaWFibGVzIHlvdSBmaW5kIGluIHRoZSBsaXRlcmF0dXJlKSBidXQgcmVtb3ZlZCB0aGlzIGFuYWx5c2lzIGJlY2F1c2UgICAKMS4gdGhlIGVzdGltYXRlcyBpbiB0aGUgbGl0ZXJhdHVyZSBhcmUgbm90IGFsbCB0aGUgc2FtZSBzdGF0aXN0aWMKMi4gZnJvbSBvdXIgZGF0YSBJIHdhcyBvbmx5IHVzaW5nIElDQydzCjMuIEkgd2FzIHNhbXBsaW5nIGZyb20gb3VyIGJvb3RzdHJhcHBlZCByZXN1bHRzIGFzIG1hbnkgZGF0YXBvaW50cyBmb3IgZWFjaCBtZWFzdXJlIGFzIHRoZXJlIGFyZSBpbiB0aGUgbGl0ZXJhdHVyZSBidXQgdGhhdCBkaWRuJ3Qgc2VlbSBsaWtlIHRoZSBiZXN0IHdheSB0byBtYWtlIHVzZSBvZiBvdXIgZGF0YSB0byBtYWtlIGl0IGNvbXBhcmFibGUgdG8gdGhlIGxpdGVyYXR1cmUuCgpEZXNwaXRlIHRoZXNlIHByb2JsZW1zIHRoZSBtYWluIHJlc3VsdCB3YXMgdGhhdCB0aGUgZWZmZWN0IHNpemUgd2FzIG11Y2ggbGFyZ2VyIGluIG91ciBkYXRhc2V0IGNvbXBhcmVkIHRvIHRoZSBsaXRlcmF0dXJlIGJ1dCBnaXZlbiB0aGUgcHJvYmxlbXMgSSB0aGluayB0aGUgc2FtcGxpbmcgYW5hbHlzZXMgYmVsb3cgYXJlIG1vcmUgaW5mb3JtYXRpdmUgdGhhbiB0aGlzLgoKV2UgYWxzbyBjaGVja2VkIHdoZXRoZXIgb3VyIHJlc3VsdHMgZGl2ZXJnZSBtb3N0IGZyb20gc3R1ZGllcyB3aXRoIHNtYWxsZXIgc2FtcGxlIHNpemVzLiBTcXVhcmUgZGlmZmVyZW5jZSBiZXR3ZWVuIG91ciBtZWFuIGVzdGltYXRlIGFuZCB0aGUgcmVsaWFiaWxpdHkgZnJvbSB0aGUgbGl0ZXJhdHVyZSBkZWNyZWFzZXMgZXhwb25lbnRpYWxseSB3aXRoIHNhbXBsZSBzaXplLiBUaGUgc21hbGxlciB0aGUgc2FtcGxlIHNpemUgaW4gdGhlIGxpdGVyYXR1cmUgdGhlIG1vcmUgdGhlIHJlbGlhYmlsaXR5IGVzdGltYXRlIGRpZmZlcnMgZnJvbSBvdXIgcmVzdWx0cy4gQnV0IHRoaXMgd2FzIGEgd2VhayByZXN1bHQgYmVjYXVzZSBtb3N0IG9mIHRoZSBzdHVkaWVzIGluIHRoZSBsaXRlcmF0dXJlIGhhdmUgc21hbGxlciBzYW1wbGUgc2l6ZXMgYW5kIHlvdSBzZWUgYm90aCBzbWFsbCBhbmQgbGFyZ2UgZGV2aWF0aW9ucyBmb3IgdGhlc2Ugc3R1ZGllcyAodGhlc2Ugd2VyZSBub3Qgc2lnbmlmaWNhbnQgZWl0aGVyKS4KCiMjIyBEaXJlY3QgcmVsYXRpb24gdG8gb3VyIHJlc3VsdHMKCkNvcnJlbGF0aW9uIGJldHdlZW4gb3VyIG1lYW4gZXN0aW1hdGVzIGZyb20gYm9vdHN0cmFwcGVkIHNhbXBsZXMgYW5kIHRoZSBsaXRlcmF0dXJlIHJldmlldyBmb3IgdGFzayB2YXJpYWJsZXMKCmBgYHtyfQpuX2RmID0gcmVsX2NvbXAgJT4lIAogIGdyb3VwX2J5KGR2KSAlPiUKICB0YWxseSgpCgpsaXRfZW1wX2NvciA9IGZ1bmN0aW9uKCl7CiAgCiAgYm9vdF9jb21wID0gZGF0YS5mcmFtZSgpCiAgCiAgZm9yKGkgaW4gMTpsZW5ndGgodW5pcXVlKHJlbF9jb21wJGR2KSkgKXsKICAgIGN1cl9kdiA9IHVuaXF1ZShyZWxfY29tcCRkdilbaV0KICAgIG4gPSBuX2RmJG5bbl9kZiRkdiA9PSBjdXJfZHZdCiAgICBzYW1wbGVfZGYgPSBib290X2RmICU+JSBmaWx0ZXIoZHYgPT0gY3VyX2R2KQogICAgdG1wID0gc2FtcGxlX24oc2FtcGxlX2RmLCBuKQogICAgYm9vdF9jb21wID0gcmJpbmQoYm9vdF9jb21wLCB0bXApCiAgfSAgCiAgCiAgcm0oY3VyX2R2LCBuLCBzYW1wbGVfZGYsIHRtcCkKICAKICAjY2hlY2sgaWYgY2JpbmQgaXMgb2sKICAjIHN1bShib290X2NvbXAkZHYgPT0gcmVsX2NvbXAkZHYpCiAgI2NiaW5kaW5nIHBlYXJzb24gYmVjYXVzZSB0aGF0IGlzIHRoZSBtb3N0IGNvbW1vbiBtZXRyaWMgaW4gdGhlIGxpdAogIHJlbF9jb21wID0gY2JpbmQocmVsX2NvbXAsIGJvb3RfY29tcCRwZWFyc29uKQogICNyZW5hbWUgbmV3IGNvbHVtbgogIG5hbWVzKHJlbF9jb21wKVt3aGljaChuYW1lcyhyZWxfY29tcCkgPT0gImJvb3RfY29tcCRwZWFyc29uIildID0gInBlYXJzb24iCiAgCiAgb3V0ID0gZGF0YS5mcmFtZSh0YXNrID0gTkEsIHN1cnZleSA9IE5BKQogIAogIG91dCR0YXNrID0gd2l0aChyZWxfY29tcCAlPiUgZmlsdGVyKHRhc2sgPT0gInRhc2siKSwgY29yKHBlYXJzb24sIHJldGVzdF9yZWxpYWJpbGl0eSkpCiAgCiAgb3V0JHN1cnZleSA9IHdpdGgocmVsX2NvbXAgJT4lIGZpbHRlcih0YXNrID09ICJzdXJ2ZXkiKSwgY29yKHBlYXJzb24sIHJldGVzdF9yZWxpYWJpbGl0eSkpCiAgCiAgcmVsX2NvbXAgPSByZWxfY29tcFssLTE2XQogIAogIHJldHVybihvdXQpCn0KCmxpdF9lbXBfY29yX291dCA9IHBseXI6OnJkcGx5KDEwMCwgbGl0X2VtcF9jb3IpCgp3cml0ZS5jc3YobGl0X2VtcF9jb3Jfb3V0LCcvVXNlcnMvemV5bmVwZW5rYXZpL0Ryb3Bib3gvUG9sZHJhY2tMYWIvU1JPX1JldGVzdF9BbmFseXNlcy9vdXRwdXQvdGFibGVzL2xpdF9lbXBfY29yX291dC5jc3YnKQoKc3VtbWFyeShsaXRfZW1wX2Nvcl9vdXQpCgpgYGAKCiMjIyBOb2lzZSBjZWlsaW5nCgpNb2RlbCBjb21wYXJpc29ucyBidWlsZGluZyBtb2RlbCB0byBwcmVkaWN0IHRoZSByZWxpYWJpbGl0aWVzIGluIHRoZSBsaXRlcmF0dXJlIGZyb20gYSBzYW1wbGUgZnJvbSBvdXIgcmVzdWx0cyB2ZXJzdXMgYSBzYW1wbGUgZm9ybSB0aGUgbGl0ZXJhdHVyZS4KCnNhbXBsZSBvbmUgcm93IHBlciBtZWFzdXJlIG91dCBvZiBsaXQgcmV2aWV3CnJeMiBvZiByZXRlc3RfcmVsaWFiaWxpdHkgfiBzYW1wbGVkX3JlbGlhYmlsaXR5IHZzLiAKcl4yIG9mIHJldGVzdF9yZWxpYWJpbGl0eSB+IG1lYW5faWNjCgpjb2VmIG9mIHJldGVzdF9yZWxpYWJpbGl0eSB+IHNhbXBsZWRfcmVsaWFiaWxpdHkgdnMuIApjb2VmIG9mIHJldGVzdF9yZWxpYWJpbGl0eSB+IG1lYW5faWNjCgpgYGB7cn0KY29tcF9saXRfcHJlZCA8LSBmdW5jdGlvbihkZil7CiAgCiAgc2FtcGxlX2Zyb21fZHYgPC0gZnVuY3Rpb24oZGYpewogICAgaWYobnJvdyhkZik+MSl7CiAgICAgIHJvd19udW0gPSBzYW1wbGUoMTpucm93KGRmKSwxKQogICAgICBzYW1wbGVfcm93ID0gZGZbcm93X251bSxdCiAgICAgIGRmID0gZGZbLXJvd19udW0sXQogICAgICBkZiRsaXRfcHJlZGljdG9yID0gc2FtcGxlX3JvdyRyZXRlc3RfcmVsaWFiaWxpdHkKICAgIH0KICAgIHJldHVybihkZikKICB9CiAgCiAgc2FtcGxlZF9kZiA9IGRmICU+JQogICAgZ3JvdXBfYnkoZHYpICU+JQogICAgZG8oc2FtcGxlX2Zyb21fZHYoLikpIAogIAogIG1vZF9saXQgPSBsbShyZXRlc3RfcmVsaWFiaWxpdHkgfiBsaXRfcHJlZGljdG9yK3NjYWxlKHNhbXBsZV9zaXplKSt0YXNrLCBkYXRhPXNhbXBsZWRfZGYpCiAgbW9kX2Jvb3QgPSBsbShyZXRlc3RfcmVsaWFiaWxpdHkgfiBtZWFuX3BlYXJzb24rc2NhbGUoc2FtcGxlX3NpemUpK3Rhc2ssIGRhdGE9c2FtcGxlZF9kZikKICAKICBvdXQgPSBkYXRhLmZyYW1lKHIyX2xpdCA9IHN1bW1hcnkobW9kX2xpdCkkci5zcXVhcmVkLAogICAgICAgICAgICAgICAgICAgcjJfYm9vdCA9IHN1bW1hcnkobW9kX2Jvb3QpJHIuc3F1YXJlZCwKICAgICAgICAgICAgICAgICAgIG1fbGl0ID0gY29lZihzdW1tYXJ5KG1vZF9saXQpKVsibGl0X3ByZWRpY3RvciIsIkVzdGltYXRlIl0sCiAgICAgICAgICAgICAgICAgICBtX2Jvb3QgPSBjb2VmKHN1bW1hcnkobW9kX2Jvb3QpKVsibWVhbl9wZWFyc29uIiwiRXN0aW1hdGUiXSkKICAKICByZXR1cm4ob3V0KQp9Cgpjb21wX2xpdF9wcmVkX291dCA9IHBseXI6OnJkcGx5KDEwMDAsIGNvbXBfbGl0X3ByZWQocmVsX2NvbXApKQpgYGAKCgpgYGB7cn0KdG1wID0gY29tcF9saXRfcHJlZF9vdXQgJT4lCiAgc2VsZWN0KC0ubikgJT4lCiAgZ2F0aGVyKGtleSwgdmFsdWUpICU+JQogIHNlcGFyYXRlKGtleSwgYygic3RhdCIsICJzYW1wbGUiKSwgc2VwID0gIl8iKQogCnRtcCRzdGF0ID0gYXMuZmFjdG9yKHRtcCRzdGF0KQpsZXZlbHModG1wJHN0YXQpIDwtIGMoImNvZWZmaWNpZW50IiwgZXhwcmVzc2lvbihyXiIyIikpCgoKdG1wICU+JSAgCiAgZ2dwbG90KGFlcyh2YWx1ZSwgZmlsbD1zYW1wbGUpKSsKICBnZW9tX2RlbnNpdHkoYWxwaGEgPSAwLjUsIHBvc2l0aW9uPSdpZGVudGl0eScsIGNvbG9yPU5BKSsKICBmYWNldF9ncmlkKC5+c3RhdCwgc2NhbGVzPSdmcmVlJywgbGFiZWxsZXIgPSBsYWJlbF9wYXJzZWQpKwogIHNjYWxlX2ZpbGxfZGlzY3JldGUoYnJlYWtzPWMoImJvb3QiLCJsaXQiKSwKICAgICAgICAgICAgICAgICAgICAgICBsYWJlbHM9YygiRW1waXJpY2FsIiwgIkxpdGVyYXR1cmUiKSwKICAgICAgICAgICAgICAgICAgICAgIG5hbWU9IlByZWRpY3Rpb24iKSsKICB4bGFiKCcnKSsKICB5bGFiKCcnKQoKZ2dzYXZlKCdMaXRfTm9pc2VfQ2VpbGluZy5qcGcnLCBkZXZpY2UgPSAianBlZyIsIHBhdGggPSAiL1VzZXJzL3pleW5lcGVua2F2aS9Ecm9wYm94L1BvbGRyYWNrTGFiL1NST19SZXRlc3RfQW5hbHlzZXMvb3V0cHV0L2ZpZ3VyZXMvIiwgd2lkdGggPSAxMCwgaGVpZ2h0ID0gNCwgdW5pdHMgPSAiaW4iLCBsaW1pdHNpemUgPSBGQUxTRSwgZHBpID0gMTAwKQpgYGAKCgpgYGB7cn0Kd2l0aChjb21wX2xpdF9wcmVkX291dCwgdC50ZXN0KHIyX2xpdCwgcjJfYm9vdCwgcGFpcmVkPVQpKQpgYGAKCmBgYHtyfQp3aXRoKGNvbXBfbGl0X3ByZWRfb3V0LCB0LnRlc3QobV9saXQsIG1fYm9vdCkpCmBgYAoKYGBge3IgZWNobz1GQUxTRX0Kcm0odG1wLCByZWxfY29tcCwgbW9kMSwgbW9kMiwgbW9kMywgbl9kZiwgaSwgbGl0X2VtcF9jb3IsIGxpdF9lbXBfY29yX291dCwgY29tcF9saXRfcHJlZCwgY29tcF9saXRfcHJlZF9vdXQpCmBgYAoKIyMgUmVsYXRpb25zaGlwIGJldHdlZW4gcmVsaWFiaWxpdHkgbWV0cmljcyAocG9pbnQgZXN0aW1hdGVzKQoKQmFzZWQgb24gW1dlaXIgKDIwMDUpXShodHRwczovL3BkZnMuc2VtYW50aWNzY2hvbGFyLm9yZy9kOTlhLzc5MGNjZTQzZjdmMjBkNzQyZjlkMzc5Yjc5ZGU0Zjc2Nzc0MC5wZGYpIElDQygzLGspIGRvZXMgbm90IHRha2UgaW4gdG8gYWNjb3VudCB3aXRoaW4gc3ViamVjdCBkaWZmZXJlbmNlcyBiZXR3ZWVuIHR3byB0aW1lIHBvaW50cyAoaS5lLiB0aGUgZml4ZWQgZWZmZWN0IG9mIHRpbWUvc3lzdGVtYXRpYyBlcnJvcikuIFRodXMsIGl0IGlzIHdlbGwgYXBwcm94aW1hdGVkIGJ5IFBlYXJzb24ncyByIGFuZCBzdWJqZWN0IHRvIHNpbWlsYXIgY3JpdGljaXNtcy4gW1dlaXIgKDIwMDUpXShodHRwczovL3BkZnMuc2VtYW50aWNzY2hvbGFyLm9yZy9kOTlhLzc5MGNjZTQzZjdmMjBkNzQyZjlkMzc5Yjc5ZGU0Zjc2Nzc0MC5wZGYpIHN1Z2dlc3RzIHJlcG9ydGluZyBhdCBsZWFzdCB0aGlzIHN5c3RlbWF0aWMgZXJyb3IgZWZmZWN0IHNpemUgaWYgb25lIGNob29zZXMgdG8gcmVwb3J0IHdpdGggSUNDKDMsaykuIEJhc2VkIG9uIGhpcyBjb25jbHVzaW9ucyBoZXJlIEkgcmVwb3J0OiAgCi0gSUNDKDMsayk6IEFzIERhdmUgY2xhcmlmaWVkIHRoaXMgcmFuZ2VzIGZyb20gMSB0byAtMS8obnVtYmVyIG9mIHJlcGVhdGVkIG1lYXN1cmVzIC0xKSBzbyBpbiBvdXIgY2FzZSB0aGlzIHJhbmdlIHdvdWxkIGJlIFstMSwgMV07IGxhcmdlciB2YWx1ZXMgd291bGQgbWVhbiB0aGF0IHRoZSB0d28gc2NvcmVzIG9mIGEgc3ViamVjdCBmb3IgYSBnaXZlbiBtZWFzdXJlIGFyZSBtb3JlIHNpbWlsYXIgdG8gZWFjaCBvdGhlciB0aGFuIHRoZXkgYXJlIHRvIHNjb3JlcyBvZiBvdGhlciBwZW9wbGUgIAotICJJQ0MgaXMgcmVmbGVjdGl2ZSBvZiB0aGUgYWJpbGl0eSBvZiBhIHRlc3QgdG8gZGlmZmVyZW50aWF0ZSBiZXR3ZWVuIGRpZmZlcmVudCBpbmRpdmlkdWFscyIKLSBwYXJ0aWFsICRcZXRhXjIkIGZvciB0aW1lICgkU1Nfe3RpbWV9L1NTX3t3aXRoaW59JCk6IGVmZmVjdCBzaXplIG9mIHRpbWUgICAKLSBTRU0gKCRcc3FydChNU197ZXJyb3J9KSQpOiBzdGFuZGFyZCBlcnJvciBvZiBtZWFzdXJlbWVudDsgdGhlIHNtYWxsZXIgdGhlIGJldHRlci4gSXQgCiJxdWFudGlmaWVzIHRoZSBwcmVjaXNpb24gb2YgaW5kaXZpZHVhbCBzY29yZXMgb24gYSB0ZXN0IiBhbmQgaXMgbm90IGRlcGVuZGVudCBvbiB0aGUgc2FtcGxlIGluIHRoZSB3YXkgSUNDIGlzIHNpbmNlIGl0IGRvZXNuJ3QgZGVwZW5kIG9uIGJldHdlZW4gc3ViamVjdCByZWxpYWJpbGl0eSAoYXQgbGVhc3QgaW4gdGhpcyBmb3JtdWxhdGlvbikgYnV0IGlzIHVuaXQtZGVwZW5kZW50LgoKV2UgY2FsY3VsYXRlZCBgciBhcy5udW1lcmljKHRhYmxlKHJlbF9kZiR0YXNrKVsxXSlgIG1lYXN1cmVzIGZvciBzdXJ2ZXlzIGFuZCBgciBhcy5udW1lcmljKHRhYmxlKHJlbF9kZiR0YXNrKVsyXSlgIG1lYXN1cmVzIGZvciBjb2duaXRpdmUgdGFza3MuICAKClRob3VnaCB3ZSBhcmUgcHJpbWFyaWx5IHJlcG9ydGluZyBJQ0MncyBhcyBvdXIgbWV0cmljIG9mIHJlbGlhYmlsaXR5IHRoZSByZXN1bHRzIGRvbid0IGNoYW5nZSBkZXBlbmRpbmcgb24gdGhlIG1ldHJpYyBjaG9zZW4uIEhlcmUgd2UgcGxvdCBwb2ludCBlc3RpbWF0ZXMgb2YgdGhyZWUgZGlmZmVyZW50IHJlbGlhYmlsaXR5IG1ldHJpY3MgYWdhaW5zdCBlYWNoIG90aGVyIChJQ0MsIFBlYXJzb24sIFNwZWFybWFuKS4gVGhlIGJvb3RzdHJhcHBlZCB2ZXJzaW9uIGlzIGVzc2VudGlhbGx5IHRoZSBzYW1lIGJ1dCB0aGUgcGxvdHMgYXJlIGJ1c2llciBkdWUgdG8gbW9yZSBkYXRhcG9pbnRzLgoKYGBge3J9CnRhYmxlKHJlbF9kZiR0YXNrKQpgYGAKCmBgYHtyIHdhcm5pbmc9RkFMU0UsIG1lc3NhZ2U9RkFMU0V9CnAxID0gcmVsX2RmICU+JQogIGdncGxvdChhZXMoc3BlYXJtYW4sIGljYywgY29sPXRhc2spKSsKICBnZW9tX3BvaW50KCkrCiAgdGhlbWVfYncoKSsKICB0aGVtZShsZWdlbmQudGl0bGUgPSBlbGVtZW50X2JsYW5rKCksCiAgICAgICAgbGVnZW5kLnBvc2l0aW9uID0gIm5vbmUiKSsKICBnZW9tX2FibGluZShpbnRlcmNlcHQgPSAwLCBzbG9wZT0xKQoKcDIgPSByZWxfZGYgJT4lCiAgZ2dwbG90KGFlcyhwZWFyc29uLCBpY2MsIGNvbD10YXNrKSkrCiAgZ2VvbV9wb2ludCgpKwogIHRoZW1lX2J3KCkrCiAgdGhlbWUobGVnZW5kLnRpdGxlID0gZWxlbWVudF9ibGFuaygpLAogICAgICAgIGxlZ2VuZC5wb3NpdGlvbiA9ICJub25lIikrCiAgZ2VvbV9hYmxpbmUoaW50ZXJjZXB0ID0gMCwgc2xvcGU9MSkKCnAzID0gcmVsX2RmICU+JQogIGdncGxvdChhZXMocGVhcnNvbiwgc3BlYXJtYW4sIGNvbD10YXNrKSkrCiAgZ2VvbV9wb2ludCgpKwogIHRoZW1lX2J3KCkrCiAgdGhlbWUobGVnZW5kLnRpdGxlID0gZWxlbWVudF9ibGFuaygpLAogICAgICAgIGxlZ2VuZC5wb3NpdGlvbiA9ICJub25lIikrCiAgZ2VvbV9hYmxpbmUoaW50ZXJjZXB0ID0gMCwgc2xvcGU9MSkKCmdyaWQuYXJyYW5nZShwMSwgcDIsIHAzLCBucm93PTEpCgpnZ3NhdmUoJ01ldHJpY19TY2F0dGVycGxvdHMuanBnJywgcGxvdCA9IGdyaWQuYXJyYW5nZShwMSwgcDIsIHAzLCBucm93PTEpLCBkZXZpY2UgPSAianBlZyIsIHBhdGggPSAiL1VzZXJzL3pleW5lcGVua2F2aS9Ecm9wYm94L1BvbGRyYWNrTGFiL1NST19SZXRlc3RfQW5hbHlzZXMvb3V0cHV0L2ZpZ3VyZXMvIiwgd2lkdGggPSAxMiwgaGVpZ2h0ID0gNCwgdW5pdHMgPSAiaW4iLCBsaW1pdHNpemUgPSBGQUxTRSwgZHBpID0gMTAwKQpgYGAKCkFzIHRoZSBzY2F0dGVyIHBsb3RzIGRlcGljdCB0aGUgY29ycmVsYXRpb25zIGJldHdlZW4gZGlmZmVyZW50IHR5cGVzIHJlbGlhYmlsaXR5IG1ldHJpY3Mgd2VyZSB2ZXJ5IGhpZ2guCgpgYGB7cn0KY29yKHJlbF9kZlssYygnc3BlYXJtYW4nLCAnaWNjJywgJ3BlYXJzb24nKV0pCmBgYAoKYGBge3IgZWNobz1GQUxTRX0Kcm0ocDEscDIscDMpCmBgYAoKTm90ZTogU29tZSB2YXJpYWJsZXMgaGF2ZSA8MCBJQ0Mncy4gVGhpcyB3b3VsZCBiZSB0aGUgY2FzZSBpZiB0aGUgJE1TX3tlcnJvcn0kPiRNU197YmV0d2Vlbn0kLiBEYXRhIGZvciB0aGVzZSB2YXJpYWJsZXMgaGF2ZSBubyByZWxhdGlvbnNoaXAgYmV0d2VlbiB0aGUgdHdvIHRpbWUgcG9pbnRzLgoKIyMgU3VtbWFyeSBvZiBhbGwgbWVhc3VyZSByZWxpYWJpbGl0aWVzCgpTdW1tYXJpemVkIGJvb3RzdHJhcHBlZCByZWxpYWJpbGl0aWVzCgpgYGB7ciBtZXNzYWdlPUZBTFNFLCB3YXJuaW5nPUZBTFNFfQpib290X2RmICU+JQogIGdyb3VwX2J5KGR2KSAlPiUKICBzdW1tYXJpc2UoaWNjX21lZGlhbiA9IHF1YW50aWxlKGljYywgcHJvYnMgPSAwLjUpLAogICAgICAgICAgICBpY2NfMi41ID0gcXVhbnRpbGUoaWNjLCBwcm9icyA9IDAuMDI1KSwKICAgICAgICAgICAgaWNjXzk3LjUgPSBxdWFudGlsZShpY2MsIHByb2JzID0gMC45NzUpLAogICAgICAgICAgICBzcGVhcm1hbl9tZWRpYW4gPSBxdWFudGlsZShzcGVhcm1hbiwgcHJvYnMgPSAwLjUpLAogICAgICAgICAgICBzcGVhcm1hbl8yLjUgPSBxdWFudGlsZShzcGVhcm1hbiwgcHJvYnMgPSAwLjAyNSksCiAgICAgICAgICAgIHNwZWFybWFuXzk3LjUgPSBxdWFudGlsZShzcGVhcm1hbiwgcHJvYnMgPSAwLjk3NSkpICU+JQogIGRhdGF0YWJsZSgpICU+JQogIGZvcm1hdFJvdW5kKGNvbHVtbnM9YygnaWNjX21lZGlhbicsICdpY2NfMi41JywgJ2ljY185Ny41JywgJ3NwZWFybWFuX21lZGlhbicsICdzcGVhcm1hbl8yLjUnLCAnc3BlYXJtYW5fOTcuNScpLCBkaWdpdHM9MykKYGBgCgpgYGB7ciBlY2hvPUZBTFNFfQptZWFzdXJlX2xhYmVscyA8LSByZWFkLmNzdignL1VzZXJzL3pleW5lcGVua2F2aS9Ecm9wYm94L1BvbGRyYWNrTGFiL1NST19SZXRlc3RfQW5hbHlzZXMvaW5wdXQvbWVhc3VyZV9sYWJlbHMuY3N2JykKbWVhc3VyZV9sYWJlbHMgPSBtZWFzdXJlX2xhYmVscyAlPiUgc2VsZWN0KC1tZWFzdXJlX2Rlc2NyaXB0aW9uKQojIENoZWNrIGlmIHRoZXJlIGFyZSBhbnkgbWlzc2luZyB2YXJpYWJsZXMKIyByZXRlc3RfcmVwb3J0X3ZhcnNbKHJldGVzdF9yZXBvcnRfdmFycyAlaW4lIG1lYXN1cmVfbGFiZWxzJGR2ID09IEZBTFNFKV0KIyBtZWFzdXJlX2xhYmVscyRkdlsobWVhc3VyZV9sYWJlbHMkZHYgJWluJSByZXRlc3RfcmVwb3J0X3ZhcnMgPT0gRkFMU0UpXQpgYGAKCmBgYHtyIHdhcm5pbmc9RkFMU0UsIG1lc3NhZ2U9RkFMU0V9CiMgRGYgd3JhbmdsaW5nIGZvciBwbG90dGluZwp0bXAgPSBtZWFzdXJlX2xhYmVscyAlPiUKICBtdXRhdGUoZHYgPSBhcy5jaGFyYWN0ZXIoZHYpKSAlPiUKICBsZWZ0X2pvaW4oYm9vdF9kZlssYygiZHYiLCAiaWNjIiwgInNwZWFybWFuIildLCBieSA9ICdkdicpIAoKdG1wID0gdG1wICU+JQogIHNlcGFyYXRlKGR2LCBjKCJ0YXNrX2dyb3VwIiwgInZhciIpLCBzZXA9IlxcLiIscmVtb3ZlPUZBTFNFLGV4dHJhPSJtZXJnZSIpICU+JQogIG11dGF0ZSh0YXNrX2dyb3VwID0gZmFjdG9yKHRhc2tfZ3JvdXAsIGxldmVscyA9IHRhc2tfZ3JvdXBbb3JkZXIodGFzayldKSkgJT4lCiAgc2VwYXJhdGUodmFyLCBjKCJ2YXIiKSwgc2VwPSJcXC4iLHJlbW92ZT1UUlVFLGV4dHJhPSJkcm9wIikgJT4lCiAgbXV0YXRlKHRhc2tfZ3JvdXAgPSBnc3ViKCJfIiwgIiAiLCB0YXNrX2dyb3VwKSwKICAgICAgICAgdmFyID0gZ3N1YigiXyIsICIgIiwgdmFyKSkgJT4lCiAgYXJyYW5nZSh0YXNrX2dyb3VwLCB2YXIpCgp0bXAgPSB0bXAgJT4lCiAgbGVmdF9qb2luKHJlbF9kZlssYygiZHYiLCAiaWNjIildLCBieSA9ICJkdiIpICU+JQogIHJlbmFtZShpY2MgPSBpY2MueCwgcG9pbnRfZXN0ID0gaWNjLnkpCgojTWFudWFsIGNvcnJlY3Rpb24KdG1wID0gdG1wICU+JQogIG11dGF0ZSh0YXNrID0gaWZlbHNlKHRhc2tfZ3JvdXAgPT0gJ2hvbHQgbGF1cnkgc3VydmV5JywgInRhc2siLCBhcy5jaGFyYWN0ZXIodGFzaykpKSAlPiUKICBtdXRhdGUodGFza19ncm91cCA9IGlmZWxzZSh0YXNrX2dyb3VwID09ICJwc3ljaG9sb2dpY2FsIHJlZnJhY3RvcnkgcGVyaW9kIHR3byBjaG9pY2VzIiwgInBzeWNob2xvZ2ljYWwgcmVmcmFjdG9yeSBwZXJpb2QiLCBpZmVsc2UodGFza19ncm91cCA9PSAiYW5nbGluZyByaXNrIHRhc2sgYWx3YXlzIHN1bm55IiwgImFuZ2xpbmcgcmlzayB0YXNrIix0YXNrX2dyb3VwKSkpICU+JQogIG11dGF0ZSh0YXNrX2dyb3VwID0gZ3N1Yigic3VydmV5IiwgIiIsIHRhc2tfZ3JvdXApKQpgYGAKClZhcmlhYmxlIGxldmVsIHN1bW1hcnkgb2YgYm9vdHN0cmFwcGVkIHJlbGlhYmlsaXRpZXMuCgpgYGB7ciB3YXJuaW5nPUZBTFNFLCBtZXNzYWdlPUZBTFNFfSAKcDQgPC0gdG1wICU+JQogIGZpbHRlcih0YXNrID09ICd0YXNrJywKICAgICAgICAgcmF3X2ZpdCA9PSAncmF3JykgJT4lCmdncGxvdChhZXMoeSA9IHZhciwgeCA9IGljYykpICsgCiAgZ2VvbV9wb2ludChjb2xvciA9ICcjMDBCRkM0JykrCiAgZ2VvbV9wb2ludChhZXMoeSA9IHZhciwgeCA9IHBvaW50X2VzdCksIGNvbG9yID0gImJsYWNrIikrCiAgZmFjZXRfZ3JpZCh0YXNrX2dyb3Vwfi4sIHN3aXRjaCA9ICJ5Iiwgc2NhbGVzID0gImZyZWVfeSIsIHNwYWNlID0gImZyZWVfeSIsIGxhYmVsbGVyID0gbGFiZWxfd3JhcF9nZW4od2lkdGg9MjApKSArCiAgdGhlbWUocGFuZWwuc3BhY2luZyA9IHVuaXQoMC43NSwgImxpbmVzIiksIAogICAgICAgIHN0cmlwLnBsYWNlbWVudCA9ICJvdXRzaWRlIiwKICAgICAgICBzdHJpcC50ZXh0LnkgPSBlbGVtZW50X3RleHQoYW5nbGU9MTgwLCBzaXplPTM2KSwKICAgICAgICBheGlzLnRleHQueSA9IGVsZW1lbnRfdGV4dChzaXplPTIwKSwKICAgICAgICBwYW5lbC5iYWNrZ3JvdW5kID0gZWxlbWVudF9yZWN0KGZpbGwgPSBOQSksCiAgICAgICAgcGFuZWwuZ3JpZC5tYWpvciA9IGVsZW1lbnRfbGluZShjb2xvdXIgPSAiZ3JleTgwIikpICsgCiAgeGxhYigiIikrCiAgeWxhYigiIikrCiAgc2NhbGVfeF9jb250aW51b3VzKGxpbWl0cyA9IGMoLTAuMjUsMSksIGJyZWFrcz1jKC0wLjI1LCAwLCAwLjI1LCAwLjUsIDAuNzUsIDEpKSsKICBzY2FsZV95X2Rpc2NyZXRlKGxhYmVscyA9IGZ1bmN0aW9uKHgpIHN0cl93cmFwKHgsIHdpZHRoID0gMTApKSsKICBnZW9tX3ZsaW5lKHhpbnRlcmNlcHQgPSAwLCBjb2xvciA9ICJyZWQiLCBzaXplID0gMSkKCnA1IDwtIHRtcCAlPiUKICBmaWx0ZXIodGFzayA9PSAnc3VydmV5JykgJT4lCmdncGxvdChhZXMoeSA9IHZhciwgeCA9IGljYykpICsgCiAgZ2VvbV9wb2ludChjb2xvciA9ICcjRjg3NjZEJykrCiAgZ2VvbV9wb2ludChhZXMoeSA9IHZhciwgeCA9IHBvaW50X2VzdCksIGNvbG9yID0gImJsYWNrIikrCiAgZmFjZXRfZ3JpZCh0YXNrX2dyb3Vwfi4sIHN3aXRjaCA9ICJ5Iiwgc2NhbGVzID0gImZyZWVfeSIsIHNwYWNlID0gImZyZWVfeSIsIGxhYmVsbGVyID0gbGFiZWxfd3JhcF9nZW4od2lkdGg9MjApKSArCiAgdGhlbWUocGFuZWwuc3BhY2luZyA9IHVuaXQoMC43NSwgImxpbmVzIiksIAogICAgICAgIHN0cmlwLnBsYWNlbWVudCA9ICJvdXRzaWRlIiwKICAgICAgICBzdHJpcC50ZXh0LnkgPSBlbGVtZW50X3RleHQoYW5nbGU9MTgwLCBzaXplPTM2KSwKICAgICAgICBheGlzLnRleHQueSA9IGVsZW1lbnRfdGV4dChzaXplPTIwKSwKICAgICAgICBwYW5lbC5iYWNrZ3JvdW5kID0gZWxlbWVudF9yZWN0KGZpbGwgPSBOQSksCiAgICAgICAgcGFuZWwuZ3JpZC5tYWpvciA9IGVsZW1lbnRfbGluZShjb2xvdXIgPSAiZ3JleTgwIikpICsgCiAgeGxhYigiIikrCiAgeWxhYigiIikrCiAgc2NhbGVfeF9jb250aW51b3VzKGxpbWl0cyA9IGMoLTAuMjUsMSksIGJyZWFrcz1jKC0wLjI1LCAwLCAwLjI1LCAwLjUsIDAuNzUsIDEpKSsKICBzY2FsZV95X2Rpc2NyZXRlKGxhYmVscyA9IGZ1bmN0aW9uKHgpIHN0cl93cmFwKHgsIHdpZHRoID0gMTApKSsKICBnZW9tX3ZsaW5lKHhpbnRlcmNlcHQgPSAwLCBjb2xvciA9ICJyZWQiLCBzaXplID0gMSkKICAKcDYgPC0gYXJyYW5nZUdyb2IocDQsIHA1LG5yb3c9MSkKCmdnc2F2ZSgnQm9vdHN0cmFwX1Jhd19WYXJfUGxvdC5qcGcnLCBwbG90ID0gcDYsIGRldmljZSA9ICJqcGVnIiwgcGF0aCA9ICIvVXNlcnMvemV5bmVwZW5rYXZpL0Ryb3Bib3gvUG9sZHJhY2tMYWIvU1JPX1JldGVzdF9BbmFseXNlcy9vdXRwdXQvZmlndXJlcy8iLCB3aWR0aCA9IDM2LCBoZWlnaHQgPSA3MiwgdW5pdHMgPSAiaW4iLCBsaW1pdHNpemUgPSBGQUxTRSwgZHBpPTUwKQoKcm0ocDQsIHA1LCBwNikKYGBgCgpgYGB7ciBlY2hvPUZBTFNFLCBvdXQud2lkdGg9JzEwMCUnfQprbml0cjo6aW5jbHVkZV9ncmFwaGljcygnL1VzZXJzL3pleW5lcGVua2F2aS9Ecm9wYm94L1BvbGRyYWNrTGFiL1NST19SZXRlc3RfQW5hbHlzZXMvb3V0cHV0L2ZpZ3VyZXMvQm9vdHN0cmFwX1Jhd19WYXJfUGxvdC5qcGcnKQpgYGAKCmBgYHtyIG1lc3NhZ2U9RkFMU0UsIHdhcm5pbmc9RkFMU0V9CnA0X3QgPC0gdG1wICU+JQogIGZpbHRlcih0YXNrID09ICd0YXNrJykgJT4lCmdncGxvdChhZXMoeCA9IGZhY3Rvcih0YXNrX2dyb3VwLCBsZXZlbHM9cmV2KHVuaXF1ZSh0YXNrX2dyb3VwKSkpLCB5ID0gaWNjKSkgKyAKICBnZW9tX3Zpb2xpbihmaWxsPScjMDBCRkM0JykrCiAgdGhlbWVfYncoKSArIAogIHRoZW1lKGF4aXMudGV4dC55ID0gZWxlbWVudF90ZXh0KHNpemU9MzApKSsKICB4bGFiKCIiKSsKICB5bGFiKCIiKSsKICBzY2FsZV95X2NvbnRpbnVvdXMobGltaXRzID0gYygtMC4yNSwxKSwgYnJlYWtzPWMoLTAuMjUsIDAsIDAuMjUsIDAuNSwgMC43NSwgMSkpKwogIHNjYWxlX3hfZGlzY3JldGUobGFiZWxzID0gZnVuY3Rpb24oeCkgc3RyX3dyYXAoeCwgd2lkdGggPSAxNSkpKwogIGdlb21faGxpbmUoeWludGVyY2VwdCA9IDAsIGNvbG9yID0gInJlZCIsIHNpemUgPSAxKSsKICBjb29yZF9mbGlwKCkKCnA1X3QgPC0gdG1wICU+JQogIGZpbHRlcih0YXNrID09ICdzdXJ2ZXknKSAlPiUKZ2dwbG90KGFlcyh4ID0gZmFjdG9yKHRhc2tfZ3JvdXAsIGxldmVscz1yZXYodW5pcXVlKHRhc2tfZ3JvdXApKSksIHkgPSBpY2MpKSArIAogIGdlb21fdmlvbGluKGZpbGw9JyNGODc2NkQnKSsKICB0aGVtZV9idygpICsgCiAgdGhlbWUoYXhpcy50ZXh0LnkgPSBlbGVtZW50X3RleHQoc2l6ZT00MykpKwogIHhsYWIoIiIpKwogIHlsYWIoIiIpKwogIHNjYWxlX3lfY29udGludW91cyhsaW1pdHMgPSBjKC0wLjI1LDEpLCBicmVha3M9YygtMC4yNSwgMCwgMC4yNSwgMC41LCAwLjc1LCAxKSkrCiAgc2NhbGVfeF9kaXNjcmV0ZShsYWJlbHMgPSBmdW5jdGlvbih4KSBzdHJfd3JhcCh4LCB3aWR0aCA9IDEwKSkrCiAgZ2VvbV9obGluZSh5aW50ZXJjZXB0ID0gMCwgY29sb3IgPSAicmVkIiwgc2l6ZSA9IDEpKwogIGNvb3JkX2ZsaXAoKQogIApwNl90IDwtIGFycmFuZ2VHcm9iKHA0X3QsIHA1X3QsbnJvdz0xKQoKZ2dzYXZlKCdCb290c3RyYXBfUG9zdGVyX1Bsb3RfdC5qcGcnLCBwbG90ID0gcDZfdCwgZGV2aWNlID0gImpwZWciLCBwYXRoID0gIi4uL291dHB1dC9maWd1cmVzLyIsIHdpZHRoID0gMjcsIGhlaWdodCA9IDQ4LCB1bml0cyA9ICJpbiIsIGxpbWl0c2l6ZSA9IEZBTFNFLCBkcGkgPSAxMDApCmBgYAoKYGBge3IgZWNobz1GQUxTRSwgb3V0LndpZHRoPScxMDAlJ30Ka25pdHI6OmluY2x1ZGVfZ3JhcGhpY3MoJy9Vc2Vycy96ZXluZXBlbmthdmkvRHJvcGJveC9Qb2xkcmFja0xhYi9TUk9fUmV0ZXN0X0FuYWx5c2VzL291dHB1dC9maWd1cmVzL0Jvb3RzdHJhcF9Qb3N0ZXJfUGxvdF90LmpwZycpCmBgYAoKYGBge3IgZWNobz1GQUxTRX0Kcm0ocDRfdCwgcDVfdCwgcDZfdCkKYGBgCgpFeGFtcGxlIG9mIHRoZSBvdmVybGF5aW5nIHByb2NlZHVyZS4KCmBgYHtyfQpwMTwtIHRtcCAlPiUKICBmaWx0ZXIoZ3JlcGwoJ3NlbGVjdGlvbl9vcHRpbWl6YXRpb24nLCBkdikpICU+JQpnZ3Bsb3QoYWVzKHggPSB2YXIsIHkgPSBpY2MpKSArIAogIGdlb21fdmlvbGluKGZpbGw9JyNGODc2NkQnKSsKICB0aGVtZV9idygpICsgCiAgdGhlbWUoYXhpcy50ZXh0LnkgPSBlbGVtZW50X3RleHQoc2l6ZT0zMCkpKwogIHhsYWIoIiIpKwogIHlsYWIoIiIpKwogIHNjYWxlX3lfY29udGludW91cyhsaW1pdHMgPSBjKC0wLjI1LDEpLCBicmVha3M9YygtMC4yNSwgMCwgMC4yNSwgMC41LCAwLjc1LCAxKSkrCiAgc2NhbGVfeF9kaXNjcmV0ZShsYWJlbHMgPSBmdW5jdGlvbih4KSBzdHJfd3JhcCh4LCB3aWR0aCA9IDE1KSkrCiAgZ2VvbV9obGluZSh5aW50ZXJjZXB0ID0gMCwgY29sb3IgPSAicmVkIiwgc2l6ZSA9IDEpKwogIGNvb3JkX2ZsaXAoKQoKcDI8LSB0bXAgJT4lCiAgZmlsdGVyKGdyZXBsKCdzZWxlY3Rpb25fb3B0aW1pemF0aW9uJywgZHYpKSAlPiUKZ2dwbG90KGFlcyh4ID0gZmFjdG9yKHRhc2tfZ3JvdXAsIGxldmVscz1yZXYodW5pcXVlKHRhc2tfZ3JvdXApKSksIHkgPSBpY2MsIGdyb3VwPWR2KSkgKyAKICBnZW9tX3Zpb2xpbihmaWxsPScjRjg3NjZEJywgcG9zaXRpb24gPSAnaWRlbnRpdHknKSsKICB0aGVtZV9idygpICsgCiAgdGhlbWUoYXhpcy50ZXh0LnkgPSBlbGVtZW50X3RleHQoc2l6ZT0zMCkpKwogIHhsYWIoIiIpKwogIHlsYWIoIiIpKwogIHNjYWxlX3lfY29udGludW91cyhsaW1pdHMgPSBjKC0wLjI1LDEpLCBicmVha3M9YygtMC4yNSwgMCwgMC4yNSwgMC41LCAwLjc1LCAxKSkrCiAgc2NhbGVfeF9kaXNjcmV0ZShsYWJlbHMgPSBmdW5jdGlvbih4KSBzdHJfd3JhcCh4LCB3aWR0aCA9IDE1KSkrCiAgZ2VvbV9obGluZSh5aW50ZXJjZXB0ID0gMCwgY29sb3IgPSAicmVkIiwgc2l6ZSA9IDEpKwogIGNvb3JkX2ZsaXAoKQoKcDM8LSB0bXAgJT4lCiAgZmlsdGVyKGdyZXBsKCdzZWxlY3Rpb25fb3B0aW1pemF0aW9uJywgZHYpKSAlPiUKZ2dwbG90KGFlcyh4ID0gZmFjdG9yKHRhc2tfZ3JvdXAsIGxldmVscz1yZXYodW5pcXVlKHRhc2tfZ3JvdXApKSksIHkgPSBpY2MpKSArIAogIGdlb21fdmlvbGluKGZpbGw9JyNGODc2NkQnLCBwb3NpdGlvbiA9ICdpZGVudGl0eScpKwogIHRoZW1lX2J3KCkgKyAKICB0aGVtZShheGlzLnRleHQueSA9IGVsZW1lbnRfdGV4dChzaXplPTMwKSkrCiAgeGxhYigiIikrCiAgeWxhYigiIikrCiAgc2NhbGVfeV9jb250aW51b3VzKGxpbWl0cyA9IGMoLTAuMjUsMSksIGJyZWFrcz1jKC0wLjI1LCAwLCAwLjI1LCAwLjUsIDAuNzUsIDEpKSsKICBzY2FsZV94X2Rpc2NyZXRlKGxhYmVscyA9IGZ1bmN0aW9uKHgpIHN0cl93cmFwKHgsIHdpZHRoID0gMTUpKSsKICBnZW9tX2hsaW5lKHlpbnRlcmNlcHQgPSAwLCBjb2xvciA9ICJyZWQiLCBzaXplID0gMSkrCiAgY29vcmRfZmxpcCgpCgpwNCA8LSBhcnJhbmdlR3JvYihwMSxwMiwgcDMsbnJvdz0xKQoKZ2dzYXZlKCdCb290c3RyYXBfRXhhbXBsZV9QbG90X3QuanBnJywgcGxvdCA9IHA0LCBkZXZpY2UgPSAianBlZyIsIHBhdGggPSAiLi4vb3V0cHV0L2ZpZ3VyZXMvIiwgd2lkdGggPSA0MSwgaGVpZ2h0ID0gNSwgdW5pdHMgPSAiaW4iLCBsaW1pdHNpemUgPSBGQUxTRSwgZHBpID0gMTAwKQpgYGAKCmBgYHtyIGVjaG89RkFMU0UsIG91dC53aWR0aD0nMTAwJSd9CmtuaXRyOjppbmNsdWRlX2dyYXBoaWNzKCcvVXNlcnMvemV5bmVwZW5rYXZpL0Ryb3Bib3gvUG9sZHJhY2tMYWIvU1JPX1JldGVzdF9BbmFseXNlcy9vdXRwdXQvZmlndXJlcy9Cb290c3RyYXBfRXhhbXBsZV9QbG90X3QuanBnJykKYGBgCgpgYGB7cn0Kcm0ocDEsIHAyLCBwMywgcDQpCmBgYAoKIyMgU3VydmV5IHZzIFRhc2tzCgpDb21wYXJpc29uIG9mIHN1cnZleSBtZWFzdXJlcyB0byBjb2duaXRpdmUgdGFzayBtZWFzdXJlcyBpbiB0aGUgYm9vdHN0cmFwcGVkIHJlc3VsdHMuIE11bHRpbGV2ZWwgbW9kZWwgd2l0aCByYW5kb20gaW50ZXJjZXB0cyBmb3IgZWFjaCBtZWFzdXJlIGFuZCBmaXhlZCBlZmZlY3Qgb2Ygc3VydmV5IHZlcnN1cyBjb2duaXRpdmUgbWVhc3VyZS4gCgpgYGB7cn0KYm9vdF9kZiA9IGJvb3RfZGYgJT4lCiAgICBtdXRhdGUodGFzayA9IGlmZWxzZShncmVwbCgic3VydmV5IixkdiksICJzdXJ2ZXkiLCJ0YXNrIiksCiAgICAgICAgICAgdGFzayA9IGlmZWxzZShncmVwbCgiaG9sdCIsZHYpLCAidGFzayIsIHRhc2spKQoKYm9vdF9kZiAlPiUKICBncm91cF9ieSh0YXNrKSAlPiUKICBzdW1tYXJpc2UoaWNjX21lZGlhbiA9IHF1YW50aWxlKGljYywgcHJvYnMgPSAwLjUpLAogICAgICAgICAgICBpY2NfMi41ID0gcXVhbnRpbGUoaWNjLCBwcm9icyA9IDAuMDI1KSwKICAgICAgICAgICAgaWNjXzk3LjUgPSBxdWFudGlsZShpY2MsIHByb2JzID0gMC45NzUpLAogICAgICAgICAgICBzcGVhcm1hbl9tZWRpYW4gPSBxdWFudGlsZShzcGVhcm1hbiwgcHJvYnMgPSAwLjUpLAogICAgICAgICAgICBzcGVhcm1hbl8yLjUgPSBxdWFudGlsZShzcGVhcm1hbiwgcHJvYnMgPSAwLjAyNSksCiAgICAgICAgICAgIHNwZWFybWFuXzk3LjUgPSBxdWFudGlsZShzcGVhcm1hbiwgcHJvYnMgPSAwLjk3NSksCiAgICAgICAgICAgIG51bV92YXJzID0gbigpLzEwMDApICU+JQogIGRhdGF0YWJsZSgpICU+JQogIGZvcm1hdFJvdW5kKGNvbHVtbnM9YygnaWNjX21lZGlhbicsICdpY2NfMi41JywgJ2ljY185Ny41JywgJ3NwZWFybWFuX21lZGlhbicsICdzcGVhcm1hbl8yLjUnLCAnc3BlYXJtYW5fOTcuNScpLCBkaWdpdHM9MykKYGBgCgoKYGBge3J9CnN1bW1hcnkobG1lclRlc3Q6OmxtZXIoaWNjIH4gIHRhc2sgKyAoMXxkdiksIGJvb3RfZGYpKQpgYGAKCiMjIyBWYXJpYW5jZSBicmVha2Rvd24KClRoZSBxdWFudGlhdGl2ZSBleHBsYW5hdGlvbiBmb3IgdGhlIGRpZmZlcmVuY2UgaW4gcmVsaWFiaWxpdHkgZXN0aW1hdGVzIGJldHdlZW4gc3VydmV5cyBhbmQgdGFza3MsIGFzIHJlY2VudGx5IGRldGFpbGVkIGJ5IEhlZGdlIGV0IGFsLiAoMjAxNyksIGxpZXMgaW4gdGhlIGRpZmZlcmVuY2UgaW4gc291cmNlcyBvZiB2YXJpYW5jZSBiZXR3ZWVuIHRoZXNlIG1lYXN1cmVzLiBTcGVjaWZpY2FsbHksIHRoZSBJQ0MgaXMgY2FsY3VsYXRlZCBhcyB0aGUgcmF0aW8gb2YgdmFyaWFuY2UgYmV0d2VlbiBzdWJqZWN0cyB2YXJpYW5jZSB0byBhbGwgc291cmNlcyBvZiB2YXJpYW5jZS4gVGh1cywgbWVhc3VyZXMgd2l0aCBoaWdoIGJldHdlZW4gc3ViamVjdHMgdmFyaWFuY2Ugd291bGQgaGF2ZSBoaWdoIHRlc3QtcmV0ZXN0IHJlbGlhYmlsaXR5LiBJbnR1aXRpdmVseSwgbWVhc3VyZXMgd2l0aCBoaWdoIGJldHdlZW4gc3ViamVjdHMgdmFyaWFuY2UgYXJlIGFsc28gYmV0dGVyIHN1aXRlZCBmb3IgaW5kaXZpZHVhbCBkaWZmZXJlbmNlIGFuYWx5c2VzIGFzIHRoZXkgd291bGQgY2FwdHVyZSB0aGUgZGlmZmVyZW5jZXMgYmV0d2VlbiB0aGUgc3ViamVjdHMgaW4gYSBzYW1wbGUuCgpIZXJlIHdlIGZpcnN0IHBsb3QgdGhlIHBlcmNlbnRhZ2Ugb2YgdmFyaWFuY2UgZXhwbGFpbmVkIGJ5IHRoZSB0aHJlZSBzb3VyY2VzIG9mIHZhcmlhbmNlIGZvciB0aGUgcG9pbnQgZXN0aW1hdGVzIG9mIG1lYXN1cmUgcmVsaWFiaWxpdGllcy4gVGhlIHBsb3Qgb25seSBpbmNsdWRlcyByYXcgbWVhc3VyZXMgKG5vIERETSBwYXJhbWV0ZXJzKSBhbmQgdGhlIG1lYXN1cmVzIGFyZSByYW5rZWQgYnkgcGVyY2VudGFnZSBvZiBiZXR3ZWVuIHN1YmplY3QgdmFyaWFiaWxpdHkgZm9yIGVhY2ggdGFzay9zdXJ2ZXkgKGkuZS4gdGhlIGJlc3QgdG8gd29yc3QgaW5kaXZpZHVhbCBkaWZmZXJlbmNlIG1lYXN1cmUgZm9yIGVhY2ggdGFzay9zdXJ2ZXkpLiBUaGVuIHdlIGNvbXBhcmUgc3RhdGlzdGljYWxseSB3aGV0aGVyIHRoZSBwZXJjZW50YWdlIG9mIHZhcmlhbmNlIGV4cGxhaW5lZCBieSB0aGVzZSBzb3VyY2VzIGRpZmZlciBiZXR3ZWVuIHRhc2tzIGFuZCBzdXJ2ZXlzLgoKYGBge3Igd2FybmluZz1GQUxTRSwgbWVzc2FnZT1GQUxTRX0KdG1wID0gcmVsX2RmICU+JQogIG11dGF0ZSh2YXJfc3Vic19wY3QgPSB2YXJfc3Vicy8odmFyX3N1YnMrdmFyX2luZCt2YXJfcmVzaWQpKjEwMCwKICAgICAgICAgdmFyX2luZF9wY3QgPSB2YXJfaW5kLyh2YXJfc3Vicyt2YXJfaW5kK3Zhcl9yZXNpZCkqMTAwLCAKICAgICAgICAgdmFyX3Jlc2lkX3BjdCA9IHZhcl9yZXNpZC8odmFyX3N1YnMrdmFyX2luZCt2YXJfcmVzaWQpKjEwMCkgJT4lCiAgc2VsZWN0KGR2LCB0YXNrLCB2YXJfc3Vic19wY3QsIHZhcl9pbmRfcGN0LCB2YXJfcmVzaWRfcGN0KSAlPiUKICBtdXRhdGUoZHYgPSBmYWN0b3IoZHYsIGxldmVscyA9IGR2W29yZGVyKHRhc2spXSkpICU+JQogIHNlcGFyYXRlKGR2LCBjKCJ0YXNrX2dyb3VwIiwgInZhciIpLCBzZXA9IlxcLiIscmVtb3ZlPUZBTFNFLGV4dHJhPSJtZXJnZSIpICU+JQogIG11dGF0ZSh0YXNrX2dyb3VwID0gZmFjdG9yKHRhc2tfZ3JvdXAsIGxldmVscyA9IHRhc2tfZ3JvdXBbb3JkZXIodGFzayldKSkgJT4lCiAgYXJyYW5nZSh0YXNrX2dyb3VwLCB2YXJfc3Vic19wY3QpICU+JQogIG11dGF0ZShyYW5rID0gcm93X251bWJlcigpKSAlPiUKICBhcnJhbmdlKHRhc2ssIHRhc2tfZ3JvdXAsIHJhbmspICU+JQogIGdhdGhlcihrZXksIHZhbHVlLCAtZHYsIC10YXNrX2dyb3VwLCAtdmFyLCAtdGFzaywgLXJhbmspICU+JQogIHVuZ3JvdXAoKSU+JQogIG11dGF0ZSh0YXNrX2dyb3VwID0gZ3N1YigiXyIsICIgIiwgdGFza19ncm91cCksCiAgICAgICAgIHZhciA9IGdzdWIoIl8iLCAiICIsIHZhcikpICU+JQogIG11dGF0ZSh0YXNrX2dyb3VwID0gaWZlbHNlKHRhc2tfZ3JvdXAgPT0gInBzeWNob2xvZ2ljYWwgcmVmcmFjdG9yeSBwZXJpb2QgdHdvIGNob2ljZXMiLCAicHN5Y2hvbG9naWNhbCByZWZyYWN0b3J5IHBlcmlvZCIsIGlmZWxzZSh0YXNrX2dyb3VwID09ICJhbmdsaW5nIHJpc2sgdGFzayBhbHdheXMgc3VubnkiLCAiYW5nbGluZyByaXNrIHRhc2siLHRhc2tfZ3JvdXApKSkgJT4lCiAgbXV0YXRlKHRhc2tfZ3JvdXAgPSBnc3ViKCJzdXJ2ZXkiLCAiIiwgdGFza19ncm91cCkpICU+JQogIGZpbHRlcih0YXNrPT0idGFzayIsCiAgICAgICAgICFncmVwbCgiRVp8aGRkbSIsIGR2KSklPiUKICBhcnJhbmdlKHRhc2tfZ3JvdXAsIHJhbmspCmxhYmVscyA9IHRtcCAlPiUKICBkaXN0aW5jdChkdiwgLmtlZXBfYWxsPVQpCgpwMSA8LSB0bXAgJT4lCiAgZ2dwbG90KGFlcyh4PWZhY3RvcihyYW5rKSwgeT12YWx1ZSwgZmlsbD1mYWN0b3Ioa2V5LCBsZXZlbHMgPSBjKCJ2YXJfcmVzaWRfcGN0IiwgInZhcl9pbmRfcGN0IiwgInZhcl9zdWJzX3BjdCIpKSkpKwogIGdlb21fYmFyKHN0YXQ9J2lkZW50aXR5JywgYWxwaGEgPSAwLjc1LCBjb2xvcj0nIzAwQkZDNCcpKwogIHNjYWxlX3hfZGlzY3JldGUoYnJlYWtzID0gbGFiZWxzJHJhbmssCiAgICAgICAgICAgICAgICAgICAgICAgbGFiZWxzID0gbGFiZWxzJHZhcikrCiAgY29vcmRfZmxpcCgpKwogIGZhY2V0X2dyaWQodGFza19ncm91cH4uLCBzd2l0Y2ggPSAieSIsIHNjYWxlcyA9ICJmcmVlX3kiLCBzcGFjZSA9ICJmcmVlX3kiKSArCiAgdGhlbWUocGFuZWwuc3BhY2luZyA9IHVuaXQoMC41LCAibGluZXMiKSwgCiAgICAgICAgc3RyaXAucGxhY2VtZW50ID0gIm91dHNpZGUiLAogICAgICAgIHN0cmlwLnRleHQueSA9IGVsZW1lbnRfdGV4dChhbmdsZT0xODApLAogICAgICAgIHBhbmVsLmJhY2tncm91bmQgPSBlbGVtZW50X3JlY3QoZmlsbCA9IE5BKSwKICAgICAgICBwYW5lbC5ncmlkLm1ham9yID0gZWxlbWVudF9saW5lKGNvbG91ciA9ICJncmV5ODUiKSwKICAgICAgICBsZWdlbmQucG9zaXRpb24gPSAnYm90dG9tJykrCiAgdGhlbWUobGVnZW5kLnRpdGxlID0gZWxlbWVudF9ibGFuaygpKSsKICBzY2FsZV9maWxsX21hbnVhbChicmVha3MgPSBjKCJ2YXJfc3Vic19wY3QiLCAidmFyX2luZF9wY3QiLCAidmFyX3Jlc2lkX3BjdCIpLAogICAgICAgICAgICAgICAgICAgICAgbGFiZWxzID0gYygiVmFyaWFuY2UgYmV0d2VlbiBpbmRpdmlkdWFscyIsCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICJWYXJpYW5jZSBiZXR3ZWVuIHNlc3Npb25zIiwKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIkVycm9yIHZhcmlhbmNlIiksCiAgICAgICAgICAgICAgICAgIHZhbHVlcz1jKCJncmV5NjUiLCAiZ3JleTQ1IiwgImdyZXkyNSIpKSsKICB5bGFiKCIiKSsKICB4bGFiKCIiKQoKdG1wID0gcmVsX2RmICU+JQogIG11dGF0ZSh2YXJfc3Vic19wY3QgPSB2YXJfc3Vicy8odmFyX3N1YnMrdmFyX2luZCt2YXJfcmVzaWQpKjEwMCwKICAgICAgICAgdmFyX2luZF9wY3QgPSB2YXJfaW5kLyh2YXJfc3Vicyt2YXJfaW5kK3Zhcl9yZXNpZCkqMTAwLCAKICAgICAgICAgdmFyX3Jlc2lkX3BjdCA9IHZhcl9yZXNpZC8odmFyX3N1YnMrdmFyX2luZCt2YXJfcmVzaWQpKjEwMCkgJT4lCiAgc2VsZWN0KGR2LCB0YXNrLCB2YXJfc3Vic19wY3QsIHZhcl9pbmRfcGN0LCB2YXJfcmVzaWRfcGN0KSAlPiUKICBtdXRhdGUoZHYgPSBmYWN0b3IoZHYsIGxldmVscyA9IGR2W29yZGVyKHRhc2spXSkpICU+JQogIHNlcGFyYXRlKGR2LCBjKCJ0YXNrX2dyb3VwIiwgInZhciIpLCBzZXA9IlxcLiIscmVtb3ZlPUZBTFNFLGV4dHJhPSJtZXJnZSIpICU+JQogIG11dGF0ZSh0YXNrX2dyb3VwID0gZmFjdG9yKHRhc2tfZ3JvdXAsIGxldmVscyA9IHRhc2tfZ3JvdXBbb3JkZXIodGFzayldKSkgJT4lCiAgYXJyYW5nZSh0YXNrX2dyb3VwLCB2YXJfc3Vic19wY3QpICU+JQogIG11dGF0ZShyYW5rID0gcm93X251bWJlcigpKSAlPiUKICBhcnJhbmdlKHRhc2ssIHRhc2tfZ3JvdXAsIHJhbmspICU+JQogIGdhdGhlcihrZXksIHZhbHVlLCAtZHYsIC10YXNrX2dyb3VwLCAtdmFyLCAtdGFzaywgLXJhbmspICU+JQogIHVuZ3JvdXAoKSU+JQogIG11dGF0ZSh0YXNrX2dyb3VwID0gZ3N1YigiXyIsICIgIiwgdGFza19ncm91cCksCiAgICAgICAgIHZhciA9IGdzdWIoIl8iLCAiICIsIHZhcikpICU+JQogIG11dGF0ZSh0YXNrX2dyb3VwID0gaWZlbHNlKHRhc2tfZ3JvdXAgPT0gInBzeWNob2xvZ2ljYWwgcmVmcmFjdG9yeSBwZXJpb2QgdHdvIGNob2ljZXMiLCAicHN5Y2hvbG9naWNhbCByZWZyYWN0b3J5IHBlcmlvZCIsIGlmZWxzZSh0YXNrX2dyb3VwID09ICJhbmdsaW5nIHJpc2sgdGFzayBhbHdheXMgc3VubnkiLCAiYW5nbGluZyByaXNrIHRhc2siLHRhc2tfZ3JvdXApKSkgJT4lCiAgbXV0YXRlKHRhc2tfZ3JvdXAgPSBnc3ViKCJzdXJ2ZXkiLCAiIiwgdGFza19ncm91cCkpICU+JQogIGZpbHRlcih0YXNrPT0ic3VydmV5IiklPiUKICBhcnJhbmdlKHRhc2tfZ3JvdXAsIHJhbmspCmxhYmVscyA9IHRtcCAlPiUKICBkaXN0aW5jdChkdiwgLmtlZXBfYWxsPVQpCgpwMiA8LSB0bXAgJT4lCiAgZ2dwbG90KGFlcyh4PWZhY3RvcihyYW5rKSwgeT12YWx1ZSwgZmlsbD1mYWN0b3Ioa2V5LCBsZXZlbHMgPSBjKCJ2YXJfcmVzaWRfcGN0IiwgInZhcl9pbmRfcGN0IiwgInZhcl9zdWJzX3BjdCIpKSkpKwogIGdlb21fYmFyKHN0YXQ9J2lkZW50aXR5JywgYWxwaGEgPSAwLjc1KSsKICBnZW9tX2JhcihzdGF0PSdpZGVudGl0eScsIGNvbG9yPScjRjg3NjZEJywgc2hvdy5sZWdlbmQ9RkFMU0UpKwogIHNjYWxlX3hfZGlzY3JldGUoYnJlYWtzID0gbGFiZWxzJHJhbmssCiAgICAgICAgICAgICAgICAgICAgICAgbGFiZWxzID0gbGFiZWxzJHZhcikrCiAgY29vcmRfZmxpcCgpKwogIGZhY2V0X2dyaWQodGFza19ncm91cH4uLCBzd2l0Y2ggPSAieSIsIHNjYWxlcyA9ICJmcmVlX3kiLCBzcGFjZSA9ICJmcmVlX3kiKSArCiAgdGhlbWUocGFuZWwuc3BhY2luZyA9IHVuaXQoMC41LCAibGluZXMiKSwgCiAgICAgICAgc3RyaXAucGxhY2VtZW50ID0gIm91dHNpZGUiLAogICAgICAgIHN0cmlwLnRleHQueSA9IGVsZW1lbnRfdGV4dChhbmdsZT0xODApLAogICAgICAgIHBhbmVsLmJhY2tncm91bmQgPSBlbGVtZW50X3JlY3QoZmlsbCA9IE5BKSwKICAgICAgICBwYW5lbC5ncmlkLm1ham9yID0gZWxlbWVudF9saW5lKGNvbG91ciA9ICJncmV5ODUiKSwKICAgICAgICBsZWdlbmQucG9zaXRpb24gPSAnYm90dG9tJykrCiAgdGhlbWUobGVnZW5kLnRpdGxlID0gZWxlbWVudF9ibGFuaygpKSsKICBzY2FsZV9maWxsX21hbnVhbChicmVha3MgPSBjKCJ2YXJfc3Vic19wY3QiLCAidmFyX2luZF9wY3QiLCAidmFyX3Jlc2lkX3BjdCIpLAogICAgICAgICAgICAgICAgICAgICAgbGFiZWxzID0gYygiVmFyaWFuY2UgYmV0d2VlbiBpbmRpdmlkdWFscyIsCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICJWYXJpYW5jZSBiZXR3ZWVuIHNlc3Npb25zIiwKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIkVycm9yIHZhcmlhbmNlIiksCiAgICAgICAgICAgICAgICAgIHZhbHVlcz1jKCJncmV5NjUiLCAiZ3JleTQ1IiwgImdyZXkyNSIpKSsKICB5bGFiKCIiKSsKICB4bGFiKCIiKQoKbXlsZWdlbmQ8LWdfbGVnZW5kKHAyKQoKcDMgPC0gYXJyYW5nZUdyb2IoYXJyYW5nZUdyb2IocDEgK3RoZW1lKGxlZ2VuZC5wb3NpdGlvbj0ibm9uZSIpLAogICAgICAgICAgICAgICAgICAgICAgICAgcDIgKyB0aGVtZShsZWdlbmQucG9zaXRpb249Im5vbmUiKSwKICAgICAgICAgICAgICAgICAgICAgICAgIG5yb3c9MSksCiAgICAgICAgICAgICBteWxlZ2VuZCwgbnJvdz0yLGhlaWdodHM9YygxMCwgMSkpCgpnZ3NhdmUoJ1ZhcmlhbmNlX0JyZWFrZG93bl9QbG90LmpwZycsIHBsb3QgPSBwMywgZGV2aWNlID0gImpwZWciLCBwYXRoID0gIi9Vc2Vycy96ZXluZXBlbmthdmkvRHJvcGJveC9Qb2xkcmFja0xhYi9TUk9fUmV0ZXN0X0FuYWx5c2VzL291dHB1dC9maWd1cmVzIiwgd2lkdGggPSAyNCwgaGVpZ2h0ID0gMjAsIHVuaXRzID0gImluIiwgZHBpID0gMTAwKQpybSh0bXAsIGxhYmVscywgcDEsIHAyICwgcDMpCmBgYAoKYGBge3IgZWNobz1GQUxTRSwgb3V0LndpZHRoPScxMDAlJ30Ka25pdHI6OmluY2x1ZGVfZ3JhcGhpY3MoJy9Vc2Vycy96ZXluZXBlbmthdmkvRHJvcGJveC9Qb2xkcmFja0xhYi9TUk9fUmV0ZXN0X0FuYWx5c2VzL291dHB1dC9maWd1cmVzL1ZhcmlhbmNlX0JyZWFrZG93bl9QbG90LmpwZycpCmBgYAoKQ29tcGFyaW5nIHR5cGVzIG9mIHZhcmlhbmNlIGZvciBzdXJ2ZXkgdnMgdGFzayBtZWFzdXJlczogU3VydmV5IG1lYXN1cmVzIGhhdmUgaGlnaGVyIGJldHdlZW4gc3ViamVjdCB2YXJpYWJpbGl0eSAgCgpOb3RlOiBUaGlzIGFuYWx5c2lzIGluY2x1ZGVzIERETSB2YXJpYWJsZXMgdG9vLgoKUnVubmluZyBzZXBhcmF0ZSBtb2RlbHMgZm9yIGRpZmZlcmVudCBzb3VyY2VzIG9mIHZhcmlhbmNlIGJlY2F1c2UgaW50ZXJhY3RpdmUgbW9kZWwgd2l0aCB2YXJpYW5jZSB0eXBlKnRhc2sgc2VlbWVkIHRvbyBjb21wbGljYXRlZC4KCkZpcnN0IHdlIGZpbmQgdGhhdCB0YXNrIG1lYXN1cmVzIGhhdmUgYSBzbWFsbGVyIHBlcmNlbnRhZ2Ugb2YgdGhlaXIgb3ZlcmFsbCB2YXJpYW5jZSBleHBsYWluZWQgYnkgdmFyaWFiaWxpdHkgYmV0d2VlbiBzdWJqZWN0cyBjb21wYXJlZCB0byBzdXJ2ZXkgbWVhc3VyZXMuCgpgYGB7cn0KdG1wID0gYm9vdF9kZiAlPiUKICBtdXRhdGUodmFyX3N1YnNfcGN0ID0gdmFyX3N1YnMvKHZhcl9zdWJzK3Zhcl9pbmQrdmFyX3Jlc2lkKSoxMDAsCiAgICAgICAgIHZhcl9pbmRfcGN0ID0gdmFyX2luZC8odmFyX3N1YnMrdmFyX2luZCt2YXJfcmVzaWQpKjEwMCwgCiAgICAgICAgIHZhcl9yZXNpZF9wY3QgPSB2YXJfcmVzaWQvKHZhcl9zdWJzK3Zhcl9pbmQrdmFyX3Jlc2lkKSoxMDApICU+JQogIHNlbGVjdChkdiwgdGFzaywgdmFyX3N1YnNfcGN0LCB2YXJfaW5kX3BjdCwgdmFyX3Jlc2lkX3BjdCkgCgpzdW1tYXJ5KGxtZXJUZXN0OjpsbWVyKHZhcl9zdWJzX3BjdH50YXNrKygxfGR2KSx0bXAlPiVzZWxlY3QoLXZhcl9pbmRfcGN0LC12YXJfcmVzaWRfcGN0KSkpCmBgYAoKV2UgYWxzbyBmaW5kIHRoYXQgYSBzaWduaWZpY2FudGx5IGxhcmdlciBwZXJjZW50YWdlIG9mIHRoZWlyIHZhcmlhbmNlIGlzIGV4cGxhaW5lZCBieSBiZXR3ZWVuIHNlc3Npb24gdmFyaWFiaWxpdHkuIExhcmdlciBiZXR3ZWVuIHNlc3Npb24gdmFyaWFiaWxpdHkgc3VnZ2VzdHMgc3lzdGVtYXRpYyBkaWZmZXJlbmNlcyBiZXR3ZWVuIHRoZSB0d28gc2Vzc2lvbnMuIFN1Y2ggc3lzdGVtYXRpYyBlZmZlY3RzIGNhbiBiZSBkdWUgdG8gZS5nLiBsZWFybmluZyBlZmZlY3RzIGFzIGV4cGxvcmVkIGxhdGVyLgoKYGBge3J9CnN1bW1hcnkobG1lclRlc3Q6OmxtZXIodmFyX2luZF9wY3R+dGFzaysoMXxkdiksdG1wJT4lc2VsZWN0KC12YXJfc3Vic19wY3QsLXZhcl9yZXNpZF9wY3QpKSkKYGBgCgpgYGB7cn0Kc3VtbWFyeShsbWVyVGVzdDo6bG1lcih2YXJfcmVzaWRfcGN0fnRhc2srKDF8ZHYpLHRtcCU+JXNlbGVjdCgtdmFyX3N1YnNfcGN0LC12YXJfaW5kX3BjdCkpKQpgYGAKCmBgYHtyfQp0bXBfc2F2ZSA9IHRtcCU+JQogIGdhdGhlcihrZXksIHZhbHVlLCAtZHYsIC10YXNrKSAlPiUKICBncm91cF9ieSh0YXNrLCBrZXkpICU+JQogIHN1bW1hcmlzZShtZWRpYW4gPSBtZWRpYW4odmFsdWUpLAogICAgICAgICAgICBzZCA9IHNkKHZhbHVlKSkgJT4lCiAgbXV0YXRlKGtleSA9IGlmZWxzZShrZXkgPT0gJ3Zhcl9pbmRfcGN0JywgJ0JldHdlZW4gc2Vzc2lvbiB2YXJpYW5jZScsIGlmZWxzZShrZXkgPT0gJ3Zhcl9zdWJzX3BjdCcsICdCZXR3ZWVuIHN1YmplY3RzIHZhcmlhbmNlJywgaWZlbHNlKGtleSA9PSAndmFyX3Jlc2lkX3BjdCcsICdSZXNpZHVhbCB2YXJpYW5jZScsTkEpKSkpICU+JQogIHJlbmFtZShNZWRpYW4gPSBtZWRpYW4sIFNEID0gc2QpICU+JQogIGFycmFuZ2UodGFzaywga2V5KQoKdG1wX3NhdmUKYGBgCgpgYGB7cn0Kc2p0LmRmKHRtcF9zYXZlICU+JSBtdXRhdGVfaWYoaXMubnVtZXJpYywgZnVucyhyb3VuZCguLCAzKSkpLCBkZXNjcmliZT1GLCBoaWRlLnByb2dyZXNzID0gVFJVRSwgc2hvdy5yb3duYW1lcyA9IEZBTFNFLCBmaWxlID0gIi9Vc2Vycy96ZXluZXBlbmthdmkvRHJvcGJveC9Qb2xkcmFja0xhYi9TUk9fUmV0ZXN0X0FuYWx5c2VzL291dHB1dC90YWJsZXMvdmFyX2JyZWFrZG93bl9zdW1tYXJ5LmRvYyIpCmBgYAoKU3VtbWFyaXppbmcgZm9yIGNsZWFyZXIgcHJlc2VudGF0aW9uLiBUaGlzIGdyYXBoIGlzIGN1cnJlbnRseSB1c2luZyB0aGUgYm9vdHN0cmFwcGVkIHJlbGlhYmlsaXRpZXMgYW5kIGlzIHRoZXJlZm9yZSBtZXNzaWVyIHRoYW4gaWYganVzdCB1c2luZyB0aGUgcG9pbnQgZXN0aW1hdGVzLgoKYGBge3J9CnRtcCAlPiUKICBnYXRoZXIoa2V5LCB2YWx1ZSwgLWR2LCAtdGFzaykgJT4lCiAgZ3JvdXBfYnkodGFzaywga2V5KSAlPiUKICBzdW1tYXJpc2UobWVhbl9wY3QgPSBtZWFuKHZhbHVlKSwKICAgICAgICAgICAgc2RfcGN0ID0gc2QodmFsdWUpLAogICAgICAgICAgICBuID0gbigpKSAlPiUKICBtdXRhdGUoY3ZsID0gcXQoMC4wMjUsIG4tMSksCiAgICAgICAgIGN2dSA9IHF0KDAuOTc1LCBuLTEpLAogICAgICAgICBjaWwgPSBtZWFuX3BjdCsoc2RfcGN0KmN2bCkvc3FydChuKSwKICAgICAgICAgY2l1ID0gbWVhbl9wY3QrKHNkX3BjdCpjdnUpL3NxcnQobiksCiAgICAgICAgIHNlbV9wY3QgPSBzZF9wY3Qvc3FydChuKSkgJT4lCiAgZ2dwbG90KGFlcyhmYWN0b3Ioa2V5LCBsZXZlbHMgPSBjKCJ2YXJfc3Vic19wY3QiLCAidmFyX2luZF9wY3QiLCAidmFyX3Jlc2lkX3BjdCIpLAogICAgICAgICAgICAgICAgICAgIGxhYmVscyA9IGMoIlZhcmlhbmNlIGJldHdlZW4gaW5kaXZpZHVhbHMiLAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIlZhcmlhbmNlIGJldHdlZW4gc2Vzc2lvbnMiLAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIkVycm9yIHZhcmlhbmNlIikpLCBtZWFuX3BjdCwgY29sb3I9dGFzaykpKwogIGdlb21fcG9pbnQocG9zaXRpb249cG9zaXRpb25fZG9kZ2Uod2lkdGggPSAwLjI1KSwgc2l6ZSA9IDQpKwogICMgZ2VvbV9lcnJvcmJhcihhZXMoeW1pbj1tZWFuX3BjdC1zZF9wY3QsIHltYXg9bWVhbl9wY3Qrc2RfcGN0KSwgcG9zaXRpb249cG9zaXRpb25fZG9kZ2Uod2lkdGggPSAwLjI1KSwgd2lkdGg9MCwgc2l6ZT0yKSsKICBnZW9tX2Vycm9yYmFyKGFlcyh5bWluPWNpbCwgeW1heD1jaXUpLCBwb3NpdGlvbj1wb3NpdGlvbl9kb2RnZSh3aWR0aCA9IDAuMjUpLCB3aWR0aD0wLjI1LCBzaXplPTIpKwogIHRoZW1lX2J3KCkrCiAgeGxhYignJykrCiAgeWxhYignUGVyY2VudCcpKwogIHRoZW1lKGxlZ2VuZC50aXRsZSA9IGVsZW1lbnRfYmxhbmsoKSwKICAgICAgICBsZWdlbmQudGV4dCA9IGVsZW1lbnRfdGV4dChzaXplPTE0KSwKICAgICAgICBsZWdlbmQucG9zaXRpb24gPSAnYm90dG9tJywKICAgICAgICBheGlzLnRleHQgPSBlbGVtZW50X3RleHQoc2l6ZT0xMiksCiAgICAgICAgYXhpcy50aXRsZS55ID0gZWxlbWVudF90ZXh0KHNpemU9MTIpKSsKICBzY2FsZV94X2Rpc2NyZXRlKGxhYmVscyA9IGZ1bmN0aW9uKHgpIHN0cl93cmFwKHgsIHdpZHRoID0gMTUpKQoKZ2dzYXZlKCdWYXJpYW5jZV9CcmVha2Rvd25fRG90UGxvdC5qcGcnLCBkZXZpY2UgPSAianBlZyIsIHBhdGggPSAiL1VzZXJzL3pleW5lcGVua2F2aS9Ecm9wYm94L1BvbGRyYWNrTGFiL1NST19SZXRlc3RfQW5hbHlzZXMvb3V0cHV0L2ZpZ3VyZXMiLCB3aWR0aCA9IDYsIGhlaWdodCA9IDUsIHVuaXRzID0gImluIiwgZHBpID0gMTAwKQpgYGAKCiMjIFN5c3RlbWF0aWMgZWZmZWN0cyBiZXR3ZWVuIHRpbWUgcG9pbnRzCgojIyMgQW5vdmEgdGltZSBlZmZlY3RzCgpUaGUgdHlwZSBvZiBJQ0Mgd2UgaGF2ZSBjaG9zZW4gZG9lcyBub3QgdGFrZSB3aXRoaW4gc3ViamVjdCAoYmV0d2VlbiBzZXNzaW9uL3N5c3RlbWF0aWMpIHZhcmlhbmNlIGluIHRvIGFjY291bnQuIFRoaXMgaXMgd2h5IFdlaXIgcmVjb21tZW5kcyBjaGVja2luZyB3aGV0aGVyIHRoZXJlIGlzIGEgc2lnbmlmaWNhbnQgY2hhbmdlIGJhc2VkIG9uIHRpbWUgYW5kIGV4YW1pbmluZyB0aGUgU0VNcy4gVGhlc2Ugc3lzdGVtYXRpYyBlZmZlY3RzIGNvdWxkIGJlIG1lYW5pbmdmdWwgYW5kIGltcG9ydGFudCB0byBhY2NvdW50IGZvciBmb3Igc29tZSBtZWFzdXJlcyAoZS5nLiB0YXNrIG1lYXN1cmVzIHRoYXQgc2hvdyBsZWFybmluZyBlZmZlY3RzKS4KCkhhZCB3ZSBjaG9zZW4gYW5vdGhlciBraW5kIG9mIElDQyB0YWtpbmcgdGhpcyBzb3VyY2Ugb2YgdmFyaWFuY2UgaW50byBhY2NvdW50IChlLmcuIDIsMSBvciAyLGspIHRoZXkgY291bGQgaGF2ZSBzdWdnZXN0ZWQgdGhhdCB0YXNrcyBoYXZlIGxvd2VyIHJlbGlhYmlsaXR5LgoKRG9pbmcgYSBzaW1wbGUgdC10ZXN0IG9uIHRoZSBkaWZmZXJlbmNlIHNjb3JlIGFsb25lIHdvdWxkIG5vdCBiZSBhIHZlcnkgcmlnb3Vyb3VzIHdheSBvZiB0ZXN0aW5nIHdoZXRoZXIgYW55IGNoYW5nZSBpcyBtZWFuaW5nZnVsIGJlY2F1c2UgdHdvIGRpc3RyaWJ1dGlvbnMgZnJvbSBib3RoIHRpbWUgcG9pbnRzIHdpdGggZXJyb3Igd291bGQgYmUgY29tcGFyZWQgdG8gZWFjaCBvdGhlci4gRm9ydHVuYXRlbHkgdGhlcmUgYXJlIHdheXMgdG8gdGFrZSB0aGUgZXJyb3IgZm9yIGJvdGggbWVhc3VyZW1lbnRzIGluIHRvIGFjY291bnQuIAoKVG8gY2hlY2sgd2hldGhlciBhIG1lYXN1cmUgc2hvd3Mgc3lzdGVtYXRpYyBkaWZmZXJlbmNlcyBiZXR3ZWVuIHRoZSB0d28gdGltZSBwb2ludHMgaW4gYSBtZWFuaW5nZnVsIG51bWJlciBvZiBib290c3RyYXBwZWQgc2FtcGxlcyB3ZSBjYW46IGNoZWNrIGlmIHRoZSBlZmZlY3Qgb2YgdGltZSBpcyBzaWduaWZpY2FudCBpbiBlYWNoIGJvb3RzdHJhcHBlZCBzYW1wbGUgYW5kIGZpbHRlciB2YXJpYWJsZXMgdGhhdCBoYXZlIG1vcmUgdGhhbiA1JSBvZiB0aGUgYm9vc3RyYXBwZWQgc2FtcGxlcyBzaG93aW5nIHNpZ25pZmljYW50IHRpbWUgZWZmZWN0cy4KCkFub3RoZXIgd2F5IG1pZ2h0IGJlIHRvIGNvbXB1dGUgY29uZmlkZW5jZSBpbnRlcnZhbHMgdXNpbmcgU0VNcyBhcyBkZXNjcmliZWQgaW4gdGhlIHNlY29uZCBoYWxmIG9mIFdlaXIgKDIwMDUpIGFuZCBjaGVjayB3aGF0IHBlcmNlbnQgb2YgcGFydGljaXBhbnRzIGhhdmUgc2NvcmVzIHRoYXQgZmFsbCBvdXQgb2YgdGhpcyByYW5nZS4gSSBoYXZlbid0IHB1cnN1ZWQgdGhpcyBmb3Igbm93LiAKCkhlcmUgd2UgYXNrOiBXaGljaCB2YXJpYWJsZXMgaGF2ZSBzaWduaWZpY2FudCB0aW1lIGVmZmVjdHMgaW4gbW9yZSB0aGFuIDUlIG9mIHRoZSBib290c3RyYXBwZWQgc2FtcGxlcz8KCjIzLzc0IHN1cnZleSBtZWFzdXJlcwoxMzMvMzcyIHRhc2sgbWVhc3VyZXMKCmBgYHtyfQpib290X2RmICU+JQogIHNlbGVjdChkdiwgcF90aW1lLCB0YXNrKSAlPiUKICBtdXRhdGUodGltZV9lZmZlY3Rfc2lnID0gaWZlbHNlKHBfdGltZTwwLjA1LDEsMCkpICU+JQogIGdyb3VwX2J5KGR2KSU+JQogIHN1bW1hcmlzZShwY3Rfc2lnX3RpbWVfZWZmZWN0ID0gc3VtKHRpbWVfZWZmZWN0X3NpZykvMTAsCiAgICAgICAgICAgIHRhc2sgPSB1bmlxdWUodGFzaykpJT4lCiAgZmlsdGVyKHBjdF9zaWdfdGltZV9lZmZlY3Q+NSkgJT4lCiAgYXJyYW5nZSh0YXNrLC1wY3Rfc2lnX3RpbWVfZWZmZWN0KSAlPiUKICB1bmdyb3VwKCklPiUKICBncm91cF9ieSh0YXNrKSAlPiUKICBzdW1tYXJpc2UoY291bnQ9bigpKQpgYGAKCkFyZSB0aGVzZSBzaWduaWZpY2FudGx5IGRpZmZlcmVudCBiZXR3ZWVuIHN1cnZleXMgYW5kIHRhc2tzPyBOby4KCmBgYHtyfQpjaGlzcS50ZXN0KG1hdHJpeChkYXRhID0gYygyMyw3NC0yMywxMzMsIDM3Mi0xMzMpLCBucm93PTIpKQpgYGAKCkFyZSB0aGV5IHByZWRvbWluYW50bHkgaW4gb25lIG9yIHRoZSBvdGhlciBkaXJlY3Rpb24gYW5kIGRvZXMgdGhhdCBkaWZmZXIgZGVwZW5kaW5nIG9uIHN1cnZleSB2cyB0YXNrPwoKYGBge3J9CmJvb3RfZGYgJT4lCiAgc2VsZWN0KGR2LCBwX3RpbWUsIHRhc2ssIHBlYXJzb24pICU+JQogIG11dGF0ZSh0aW1lX2VmZmVjdF9zaWcgPSBpZmVsc2UocF90aW1lPDAuMDUsMSwwKSkgJT4lCiAgZ3JvdXBfYnkoZHYpJT4lCiAgc3VtbWFyaXNlKHBjdF9zaWdfdGltZV9lZmZlY3QgPSBzdW0odGltZV9lZmZlY3Rfc2lnKS8xMCwKICAgICAgICAgICAgdGFzayA9IHVuaXF1ZSh0YXNrKSwKICAgICAgICAgICAgbWVhbl9wZWFyc29uID0gbWVhbihwZWFyc29uKSklPiUKICBmaWx0ZXIocGN0X3NpZ190aW1lX2VmZmVjdD41KSAlPiUKICBhcnJhbmdlKHRhc2ssLXBjdF9zaWdfdGltZV9lZmZlY3QpICU+JQogIGFycmFuZ2UobWVhbl9wZWFyc29uKSAKICAKYGBgCgojIyMgU0VNIENJIGNhbGN1bGF0aW9uCgpTdGVwIDE6IFQgPSBHcmFuZCBtZWFuICsgSUNDICogKFN1YmplY3Qgc2NvcmUgLSBHcmFuZCBtZWFuKQpTdGVwIDI6IFNFUCA9IFNEIG9mIGJvdGggbWVhc3VyZW1lbnRzICogc3FydCgxLUlDQ14yKQoKSGVyZSBJIGNhbGN1bGF0ZSB0aGUgcHJvcG9ydGlvbiBvZiBzdWJqZWN0cyB0aGF0IG1vdmUgbW9yZSB0aGFuIG9uZSBzdGFuZGFyZCBlcnJvciBvZiBwcmVkaWN0aW9uIChTRVApIGluIHQyIGNvbXBhcmVkIHRvIHQxIGZvciBlYWNoIG1lYXN1cmUuIAoKVGhpcyBpcyBvZGQgdG8gdGhpbmsgYWJvdXQgYmVjYXVzZSB0aGUgbGFyZ2VyIHRoZSBJQ0Mgb2YgYSBtZWFzdXJlIHRoZSBzbWFsbGVyIHRoZSBTRVAuIFNvIHZlcnkgc21hbGwgZGlmZmVyZW5jZXMgYmV0d2VlbiB0aGUgdHdvIHRpbWUgcG9pbnRzIGNhbiBiZSBjYXRlZ29yaXplZCBhcyAnbWVhbmluZ2Z1bCcgYmFzZWQgb24gdGhlIHRpbnkgU0VQLgoKV2hhdCB5b3UgYXJlIGludGVyZXN0ZWQgaW4gaXMgbm90IG5lY2Vzc2FyaWx5IHdoZXRoZXIgaW5kaXZpZHVhbHMgY2hhbmdlIGF0IGFsbCBiZXR3ZWVuIHRoZSB0d28gdGltZSBwb2ludHMgdGhvdWdoLiBZb3Ugd2FudCB0byBrbm93IGlmIHRoaXMgY2hhbmdlIGlzIHN5c3RlbWF0aWMgaW4gb25lIGRpcmVjdGlvbi4KCkhlcmUgSSBjYWxjdWxhdGUgdGhlIHByb3BvcnRpb24gb2YgcGVvcGxlIHNob3dpbmcgJ21lYW5pbmdmdWwnIGNoYW5nZXMgaW4gb25lIGRpcmVjdGlvbiBvciB0aGUgb3RoZXIuIFRvIGludGVncmF0ZSBib3RoIG9mIHRoZXNlIGRpcmVjdGlvbiBJIHN1YnRyYWN0ZWQgb25lIGZyb20gdGhlIG90aGVyIGFuZCBmaWx0ZXJlZCB0aGUgdmFyaWFibGVzIHRoYXQgaGF2ZSBtb3JlIHRoYW4gNSUgb2YgdGhlIHBhcnRpY2lwYW50cyBzaG93aW5nIG1lYW5pbmdmdWwgY2hhbmdlIGluIG9uZSBkaXJlY3Rpb24gb3ZlciB0aGUgb3RoZXIgKHNvIGlmIGEgdmFyaWFibGUgaGFzIDEwIHBhcnRpY2lwYW50cyBzaG93aW5nIGRpZmZlcmVuY2UgaW4gb25lIGFuZCAxMCBpbiB0aGUgb3RoZXIgZGlyZWN0aW9uIHRoaXMgd291bGQgY2FuY2VsIG91dCBidXQgYSB2YXJpYWJsZXMgd2l0aCAxNSBwZW9wbGUgc2hvd2luZyBhIHBvc2l0aXZlIGFuZCA1IG5lZ2F0aXZlIGNoYW5nZSB3b3VsZCByZW1haW4pLgoKCmBgYHtyfQpnZXRfaW5kX2NpID0gZnVuY3Rpb24oZHZfdmFyKXsKICBtYXRjaGVkID0gbWF0Y2hfdDFfdDIoZHZfdmFyKQogIGdyYW5kX21lYW4gPSBtZWFuKG1hdGNoZWQkc2NvcmUpCiAgZ3JhbmRfc2QgPSBzZChtYXRjaGVkJHNjb3JlKQogIGR2X2ljYyA9IGdldF9pY2MoZHZfdmFyKQogIHNlcCA9IGdyYW5kX3NkICogc3FydCgxLShkdl9pY2NeMikpCiAgbWF0Y2hlZCA9IG1hdGNoZWQgJT4lIAogICAgc3ByZWFkKHRpbWUsIHNjb3JlKSAlPiUKICAgIHJlbmFtZSgidDEiPSIxIiwgInQyIj0iMiIpICU+JQogICAgbXV0YXRlKHRydWVfc2NvcmUgPSBncmFuZF9tZWFuKyhkdl9pY2MqKHQxLWdyYW5kX21lYW4pKSwKICAgICAgICAgICBjaV91cCA9IHRydWVfc2NvcmUrKDEuOTYqc2VwKSwKICAgICAgICAgICBjaV9sb3cgPSB0cnVlX3Njb3JlLSgxLjk2KnNlcCksCiAgICAgICAgICAgdDJfYWJvdmVfY2kgPSBpZmVsc2UodDI+Y2lfdXAsMSwwKSwKICAgICAgICAgICB0Ml9iZWxvd19jaSA9IGlmZWxzZSh0MjxjaV9sb3csMSwgMCkpCiAgICByZXR1cm4obWF0Y2hlZCkKfQoKZ2V0X3Byb3Bfb3V0X2NpID0gZnVuY3Rpb24oZHZfdmFyKXsKICBnZXRfaW5kX2NpKGR2X3ZhcikgJT4lCiAgICBzdW1tYXJpc2UocHJvcF9hYm92ZV9jaSA9IHN1bSh0Ml9hYm92ZV9jaSkvbigpLAogICAgICAgICAgICAgIHByb3BfYmVsb3dfY2kgPSBzdW0odDJfYmVsb3dfY2kvbigpKSkKfQoKI0NyZWF0ZSBkZiBvZiBwb2ludCBlc3RpbWF0ZSByZWxpYWJpbGl0aWVzCmluZF9jaV9kZiA8LSBkYXRhLmZyYW1lKHByb3BfYWJvdmVfY2kgPSByZXAoTkEsIGxlbmd0aChudW1lcmljX2NvbHMpKSwKICAgICAgICAgICAgICAgICAgICAgICAgcHJvcF9iZWxvd19jaSA9IHJlcChOQSwgbGVuZ3RoKG51bWVyaWNfY29scykpKQoKcm93Lm5hbWVzKGluZF9jaV9kZikgPC0gbnVtZXJpY19jb2xzCgpmb3IoaSBpbiAxOmxlbmd0aChudW1lcmljX2NvbHMpKXsKICAgIGluZF9jaV9kZltudW1lcmljX2NvbHNbaV0sICdwcm9wX2Fib3ZlX2NpJ10gPC0gZ2V0X3Byb3Bfb3V0X2NpKG51bWVyaWNfY29sc1tpXSkkcHJvcF9hYm92ZV9jaSAKICAgIGluZF9jaV9kZltudW1lcmljX2NvbHNbaV0sICdwcm9wX2JlbG93X2NpJ10gPC0gZ2V0X3Byb3Bfb3V0X2NpKG51bWVyaWNfY29sc1tpXSkkcHJvcF9iZWxvd19jaSAKfQoKaW5kX2NpX2RmJGR2ID0gcm93Lm5hbWVzKGluZF9jaV9kZikKcm93Lm5hbWVzKGluZF9jaV9kZikgPSBzZXEoMTpucm93KGluZF9jaV9kZikpCmluZF9jaV9kZiR0YXNrID0gJ3Rhc2snCmluZF9jaV9kZltncmVwKCdzdXJ2ZXknLCBpbmRfY2lfZGYkZHYpLCAndGFzayddID0gJ3N1cnZleScKaW5kX2NpX2RmW2dyZXAoJ2hvbHQnLCBpbmRfY2lfZGYkZHYpLCAndGFzayddID0gInRhc2siCmluZF9jaV9kZiA9IGluZF9jaV9kZiAlPiUKICBzZWxlY3QoZHYsIHRhc2ssIHByb3BfYWJvdmVfY2ksIHByb3BfYmVsb3dfY2kpCgptZWFuX2RpZmZfZGYgPSBpbmRfY2lfZGYgJT4lCiAgbXV0YXRlKHByb3Bfb25lX2RpcmVjdGlvbiA9IHByb3BfYWJvdmVfY2ktcHJvcF9iZWxvd19jaSkgJT4lCiAgZmlsdGVyKGFicyhwcm9wX29uZV9kaXJlY3Rpb24pPjAuMDUpICU+JQogIGFycmFuZ2UoLWFicyhwcm9wX29uZV9kaXJlY3Rpb24pKQoKbWVhbl9kaWZmX2RmCgojIHdyaXRlLmNzdihtZWFuX2RpZmZfZGYsICcvVXNlcnMvemV5bmVwZW5rYXZpL0Ryb3Bib3gvUG9sZHJhY2tMYWIvU1JPX1JldGVzdF9BbmFseXNlcy9vdXRwdXQvdGFibGVzL21lYW5fZGlmZl9kZi5jc3YnKQpgYGAKClRoaXMgZG9lc24ndCBtYWtlIGl0IGVhc2llciB0byBpbnRlcnByZXQgd2hldGhlciBpdCBpcyBhIHBlcmZvcm1hbmNlIGltcHJvdmVtZW50IG9yIGRlY2xpbmUuIEluIGZhY3QgJ3BlcmZvcm1hbmNlJyBpc24ndCBldmVuIG5lY2Vzc2FyaWx5IHRoZSBjb3JyZWN0IHRlcm0gaGVyZS4gVXNpbmcgdGhlIHZhbGVuY2UgYXNzaWdubWVudHMgb2YgdGhlIG1lYXN1cmVzIGxvb2sgYXQgd2hldGhlciBpdCB0cmFuc2xhdGVzIHRvIGFuIGluY3JlYXNlIG9yIGRlY3JlYXNlIGluIHNlbGYtcmVndWxhdGlvbi4gT2YgdGhlIDY2IHRoaXMgbWFrZXMgc2Vuc2UgZm9yIDU0IHZhcmlhYmxlcy4KCmBgYHtyIHdhcm5pbmc9RkFMU0UsIG1lc3NhZ2U9RkFMU0V9Cm1lYW5fZGlmZl9kZiA9IHJlYWQuY3N2KCcvVXNlcnMvemV5bmVwZW5rYXZpL0Ryb3Bib3gvUG9sZHJhY2tMYWIvU1JPX1JldGVzdF9BbmFseXNlcy9vdXRwdXQvdGFibGVzL21lYW5fZGlmZl9kZi5jc3YnKQoKdGFibGUoYXMuY2hhcmFjdGVyKG1lYW5fZGlmZl9kZiR2YWxlbmNlKSwgdXNlTkEgPSAiaWZhbnkiKQoKbWVhbl9kaWZmX2RmICU+JQogIG11dGF0ZShzY19pbmNyZWFzZSA9IGlmZWxzZShwcm9wX29uZV9kaXJlY3Rpb24+MCZ2YWxlbmNlPT0iUG9zIiwxLGlmZWxzZShwcm9wX29uZV9kaXJlY3Rpb248MCZ2YWxlbmNlPT0iTmVnIiwxLDApKSwKICAgICAgICAgc2NfZGVjcmVhc2UgPSBpZmVsc2UocHJvcF9vbmVfZGlyZWN0aW9uPjAmdmFsZW5jZT09Ik5lZyIsMSxpZmVsc2UocHJvcF9vbmVfZGlyZWN0aW9uPDAmdmFsZW5jZT09IlBvcyIsMSwwKSkpICU+JQpzdW1tYXJpc2Uoc3VtX3NjX2luY3JlYXNlID0gc3VtKHNjX2luY3JlYXNlLG5hLnJtPVQpLAogICAgICAgICAgc3VtX3NjX2RlY3JlYXNlID0gc3VtKHNjX2RlY3JlYXNlLG5hLnJtPVQpKQoKYGBgCgpWYXJpYWJsZXMgdGhhdCBzaG93IG1vcmUgdGhhbiA1JSBkaWZmZXJlbmNlIGluIGEgZGlyZWN0aW9uIGRvIG5vdCBkZXBlbmQgb24gd2hldGhlciB0aGV5IHRyYW5zbGF0ZSB0byBhIHNlbGYgY29udHJvbCBpbmNyZWFzZSBvciBkZWNyZWFzZS4KCmBgYHtyfQpjaGlzcS50ZXN0KG1hdHJpeChkYXRhID0gYygyMyw1NC0yMywzMSw1NC0zMSksIG5yb3c9MikpCmBgYAoKRG8gdGhleSBkaWZmZXIgZGVwZW5kaW5nIG9uIHdoZXRoZXIgdGhleSBhcmUgdGFzayBvciBzdXJ2ZXkgdmFyaWFibGVzPyBOby4KCmBgYHtyfQpjaGlzcS50ZXN0KG1hdHJpeChkYXRhID0gYyg1LDc0LTUsNjEsMzcyLTYxKSwgbnJvdz0yKSkKYGBgCgpgYGB7cn0Kcm0obWVhbl9kaWZmX2RmKQpgYGAKCiMjIFRhc2sgUmVsaWFiaWxpdGllcwoKSGVyZSB3ZSBzdW1tYXJpemUgdGhlIHJlc3VsdHMgb24gYSB0YXNrIGxldmVsIHRvIG1ha2UgaXQgbW9yZSBkaWdlc3RhYmxlIGFuZCBlYXNpZXIgdG8gbWFrZSBjb250YWN0IHdpdGggdGhlIGxpdGVyYXR1cmUuICAKCldlIHJlZHVjZSB0aGUgbGlzdCBvZiB0YXNrIG1lYXN1cmVzIHRvIGEgbGlzdCBvZiBvbmUgcGVyIHRhc2sgYnkgYXZlcmFnaW5nIG9ubHkgdGhlIHJhdyBtZWFzdXJlcyBmcm9tIGFsbCB0aGUgdHJpYWxzIGluIGEgdGFzay4gV2UgY2hvc2UgdG8gcmVkdWNlIHRoZSBpbmZvcm1hdGlvbiBpbiB0aGlzIG1hbm5lciB0byBhdm9pZCBhbnkgYmlhcyBzdGVtbWluZyBmcm9tIGRpZmZlcmVudGlhbCBhbW91bnQgb2YgaW50ZXJlc3QgYW5kIHByb2NlZHVyZXMgYXBwbGllZCB0byBjZXJ0YWluIHRhc2tzIG92ZXIgb3RoZXJzIChlLmcuIGEgdGFzayBjYW4gaGF2ZSBvdmVyIDEwIG1lYXN1cmVzIGJlY2F1c2UgaXQgaGFzIG11bHRpcGxlIGNvbmRpdGlvbnMgYW5kIHdlIGhhdmUgY2hvc2VuIHRvIGZpdCBERE0ncyBmb3Igc3BlY2lmaWMgY29uZGl0aW9ucyB3aGlsZSBhbm90aGVyIG1pZ2h0IG9ubHkgaGF2ZSAyIGR1ZSB0byBvdXIgcmVsYXRpdmUgaW5leHBlcmllbmNlIGFuZCBsYWNrIG9mIGludGVyZXN0IGluIGl0KS4gV2UgY2hlY2sgd2hldGhlciB0aGUgbnVtYmVyIG9mIHRyaWFscyBpbiBhIHRhc2sgaGFzIGEgc2lnbmlmaWNhbnQgZWZmZWN0IG9uIHRoZXNlIGF2ZXJhZ2UgcmVsaWFiaWxpdGllcyBvZiByYXcgbWVhc3VyZXMgYXMgd2VsbC4gCgpXZSBmaWx0ZXIgb3V0IHRoZSBERE0gcGFyYW1ldGVycyBhbmQgbWVhc3VyZXMgZm9yIHNwZWNpZmljIGNvbnRyYXN0cy4gTm90ZSB0aGF0IHRoaXMgZG9lcyBsZWF2ZSBzb21lIHRhc2tzIHdpdGggbWVhc3VyZXMgdGhhdCBhcmUgbW9kZWwgZml0cyBhbmQvb3IgZm9yIHNwZWNpZmljIGNvbmRpdGlvbnMgKGJlY2F1c2UgYXQgbGVhc3QgdGhlIGN1cnJlbnQgZGF0YXNldHMgZG8gbm90IGluY2x1ZGUgbWVhc3VyZXMgdGhhdCBhcmUgYmFzZWQgb24gYWxsIHRoZSB0cmlhbHMgKiphbmQqKiBhcmUgcmF3IHRob3VnaCBJIGNvdWxkIGRpdmUgaW4gdG8gdmFyaWFibGVzX2V4aGF1c3RpdmUgZm9yIHN1Y2ggbWVhc3VyZXMuIEZvciBleGFtcGxlIHRoZSBhdmVyYWdlIHJlbGlhbGliaWxpdHkgZm9yIEtpcmJ5IGlzIGJhc2VkIG9uIHRocmVlIGRpc2NvdW50IHJhdGVzIGZvciBzcGVjaWZpYyBjb25kaXRpb25zLikuIEhlcmUncyB0aGUgb3JkZXIgb2YgdGFza3MgYnkgbWVhbiByZWxpYWJpbGl0eSBzb3J0ZWQgZm9yIElDQyBhbmQgdGhlbiBTcGVhcm1hbidzICRccmhvJC4KCmBgYHtyIHdhcm5pbmc9RkFMU0UsIG1lc3NhZ2U9RkFMU0V9CnRtcCA9IG1lYXN1cmVfbGFiZWxzICU+JQogIG11dGF0ZShkdiA9IGFzLmNoYXJhY3RlcihkdikpICU+JQogIGZpbHRlcih0YXNrID09ICd0YXNrJykgJT4lCiAgbGVmdF9qb2luKGJvb3RfZGZbLGMoImR2IiwgImljYyIsICJzcGVhcm1hbiIpXSwgYnkgPSAnZHYnKSAlPiUKICBmaWx0ZXIob3ZlcmFsbF9kaWZmZXJlbmNlICE9ICdkaWZmZXJlbmNlJyAmIHJhd19maXQgJWluJSBjKCdFWicsICdoZGRtJykgPT0gRkFMU0UpJT4lCiAgc2VwYXJhdGUoZHYsIGMoJ3Rhc2tfbmFtZScsICdleHRyYV8xJywgJ2V4dHJhXzInKSwgc2VwID0gJ1xcLicscmVtb3ZlPUZBTFNFKSAlPiUKICBzZWxlY3QoLWV4dHJhXzEsIC1leHRyYV8yKSAlPiUKICBncm91cF9ieSh0YXNrX25hbWUpICU+JQogIHN1bW1hcmlzZShtZWRpYW5faWNjID0gbWVkaWFuKGljYyksCiAgICAgICAgICAgIG1lZGlhbl9zcGVhcm1hbiA9IG1lZGlhbihzcGVhcm1hbiksCiAgICAgICAgICAgIGljY18yLjUgPSBxdWFudGlsZShpY2MsIHByb2JzID0gMC4wMjUpLAogICAgICAgICAgICBpY2NfOTcuNSA9IHF1YW50aWxlKGljYywgcHJvYnMgPSAwLjk3NSksCiAgICAgICAgICAgIG1pbl9zcGVhcm1hbiA9IG1pbihzcGVhcm1hbiksCiAgICAgICAgICAgIG1heF9zcGVhcm1hbiA9IG1heChzcGVhcm1hbiksCiAgICAgICAgICAgIG51bV9tZWFzdXJlcyA9IG4oKS8xMDAwLAogICAgICAgICAgICBtZWFuX251bV90cmlhbHMgPSByb3VuZChtZWFuKG51bV9hbGxfdHJpYWxzKSkpJT4lCiAgYXJyYW5nZSgtbWVkaWFuX2ljYywgLW1lZGlhbl9zcGVhcm1hbikKCnRtcCAlPiUKICBkYXRhdGFibGUoKSAlPiUKICBmb3JtYXRSb3VuZChjb2x1bW5zPWMoJ21lZGlhbl9zcGVhcm1hbicsICdtZWRpYW5faWNjJywKICAgICAgICAgICAgICAgICAgICAgICAgJ21pbl9zcGVhcm1hbicsICdpY2NfMi41JywKICAgICAgICAgICAgICAgICAgICAgICAgJ21heF9zcGVhcm1hbicsICdpY2NfOTcuNScpLCBkaWdpdHM9MykKYGBgCgoKYGBge3J9CnRtcCA9IHRtcCU+JQogIHNlbGVjdCgtbWVkaWFuX3NwZWFybWFuLCAtbWluX3NwZWFybWFuLCAtbWF4X3NwZWFybWFuKSAlPiUKICBtdXRhdGUodGFza19uYW1lID0gZ3N1YigiXyIsICIgIiwgdGFza19uYW1lKSwKICAgICAgICAgdGFza19uYW1lID0gZ3N1YigiKF58W1s6c3BhY2U6XV0pKFtbOmFscGhhOl1dKSIsICJcXDFcXFVcXDIiLCB0YXNrX25hbWUsIHBlcmw9VFJVRSkpCgpuYW1lcyh0bXApID0gZ3N1YigiXyIsICIgIiwgbmFtZXModG1wKSkKbmFtZXModG1wKSA9IGdzdWIoIihefFtbOnNwYWNlOl1dKShbWzphbHBoYTpdXSkiLCAiXFwxXFxVXFwyIiwgbmFtZXModG1wKSwgcGVybD1UUlVFKQoKc2p0LmRmKHRtcCAlPiUgbXV0YXRlX2lmKGlzLm51bWVyaWMsIGZ1bnMocm91bmQoLiwgMykpKSwgZGVzY3JpYmU9RiwgaGlkZS5wcm9ncmVzcyA9IFRSVUUsIHNob3cucm93bmFtZXMgPSBGQUxTRSwgZmlsZSA9ICIvVXNlcnMvemV5bmVwZW5rYXZpL0Ryb3Bib3gvUG9sZHJhY2tMYWIvU1JPX1JldGVzdF9BbmFseXNlcy9vdXRwdXQvdGFibGVzL3Rhc2tfcmVsX3RhYmxlLmRvYyIpCmBgYAoKIyMjIE51bWJlciBvZiB0cmlhbHMKCkRvZXMgbnVtYmVyIG9mIGl0ZW1zIGluIGEgdGFzayBoYXZlIGEgc2lnbmlmaWNhbnQgZWZmZWN0IG9uIHRoZSBhdmVyYWdlIElDQyBvZiAobW9zdGx5KSByYXcgbWVhc3VyZXMgZm9yIGFsbCB0cmlhbHMgZnJvbSBhIHRhc2s/IE5vLiAobm8gZWZmZWN0IG9uIFNwZWFybWFuIGVpdGhlcikKCmBgYHtyIHdhcm5pbmc9RkFMU0UsIG1lc3NhZ2U9RkFMU0V9CnRtcCA9IG1lYXN1cmVfbGFiZWxzICU+JQogIG11dGF0ZShkdiA9IGFzLmNoYXJhY3RlcihkdikpICU+JQogIGZpbHRlcih0YXNrID09ICd0YXNrJykgJT4lCiAgIyBsZWZ0X2pvaW4ocmVsX2RmWyxjKCJkdiIsICJzcGVhcm1hbiIsImljYyIpXSwgYnk9J2R2JykgJT4lCiAgbGVmdF9qb2luKGJvb3RfZGZbLGMoImR2IiwgInNwZWFybWFuIiwiaWNjIildLCBieT0nZHYnKSAlPiUKICBmaWx0ZXIob3ZlcmFsbF9kaWZmZXJlbmNlICE9ICdkaWZmZXJlbmNlJyAmIHJhd19maXQgJWluJSBjKCdFWicsICdoZGRtJykgPT0gRkFMU0UpJT4lCiAgc2VwYXJhdGUoZHYsIGMoJ3Rhc2tfbmFtZScsICdleHRyYV8xJywgJ2V4dHJhXzInKSwgc2VwID0gJ1xcLicscmVtb3ZlPUZBTFNFKSAlPiUKICBzZWxlY3QoLWV4dHJhXzEsIC1leHRyYV8yKQoKIyBzdW1tYXJ5KGxtKGljYyB+IG51bV9hbGxfdHJpYWxzLCBkYXRhID0gdG1wKSkKc3VtbWFyeShsbWVyVGVzdDo6bG1lcihpY2MgfiBudW1fYWxsX3RyaWFscyArICgxfGR2KSwgZGF0YSA9IHRtcCkpCmBgYAoKYGBge3Igd2FybmluZz1GQUxTRSwgbWVzc2FnZT1GQUxTRX0KbWVhc3VyZV9sYWJlbHMgJT4lCiAgbXV0YXRlKGR2ID0gYXMuY2hhcmFjdGVyKGR2KSkgJT4lCiAgZmlsdGVyKHRhc2sgPT0gJ3Rhc2snKSAlPiUKICAjIGxlZnRfam9pbihyZWxfZGZbLGMoImR2IiwgInNwZWFybWFuIiwiaWNjIildLCBieT0nZHYnKSAlPiUKICBsZWZ0X2pvaW4oYm9vdF9kZlssYygiZHYiLCAic3BlYXJtYW4iLCJpY2MiKV0sIGJ5PSdkdicpICU+JQogIGZpbHRlcihvdmVyYWxsX2RpZmZlcmVuY2UgIT0gJ2RpZmZlcmVuY2UnICYgcmF3X2ZpdCAlaW4lIGMoJ0VaJywgJ2hkZG0nKSA9PSBGQUxTRSklPiUKICBzZXBhcmF0ZShkdiwgYygndGFza19uYW1lJywgJ2V4dHJhXzEnLCAnZXh0cmFfMicpLCBzZXAgPSAnXFwuJyxyZW1vdmU9RkFMU0UpICU+JQogIHNlbGVjdCgtZXh0cmFfMSwgLWV4dHJhXzIpICU+JQogIGdncGxvdChhZXMobnVtX2FsbF90cmlhbHMsIGljYykpKwogIGdlb21fcG9pbnQoKSsKICBnZW9tX3Ntb290aChtZXRob2Q9ImxtIikrCiAgdGhlbWVfYncoKSsKICB4bGFiKCJOdW1iZXIgb2YgdHJpYWxzIikrCiAgeWxhYigiSUNDIikKYGBgCgojIyMjIFRyaWFsIG51bWJlciBkZXBlbmRlbmNlIGludHJhbWVhc3VyZSAKClRoZSBhYm92ZSBhbmFseXNpcyB3YXMgbG9va2luZyBhdCB0aGUgZWZmZWN0IG9mIG51bWJlciBvZiB0cmlhbHMgYWNyb3NzIHRhc2tzLiBCdXQgc29tZSB0YXNrcyBtaWdodCBiZSBiYWQgZm9yIGluZGl2aWR1YWwgZGlmZmVyZW5jZSBtZWFzdXJlbWVudCByZWdhcmRsZXNzIG9mIGhvdyBtYW55IHRyaWFscyB0aGVyZSBhcmUgaW4gdGhlbSB3aGVyZWFzIGZvciBvdGhlcnMgZmV3ZXIgdHJpYWxzIG1pZ2h0IGJlIHlpZWxkaW5nIGEgc3VmZmljaWVudGx5IHJlbGlhYmxlIG1lYXN1cmUuCgpGb3IgdGFza3MgZm9yIHdoaWNoIGRlcGVuZGVudCB2YXJpYWJsZXMgYXJlIGVzdGltYXRlZCB1c2luZyBtYW55IHRyaWFscyBvbmUgY2FuIGFzazogRG9lcyB0aGUgc2FtZSBtZWFzdXJlIGdldCBsZXNzIHJlbGlhYmxlIGlmIGZld2VyIHRyaWFscyBhcmUgdXNlZCB0byBlc3RpbWF0ZSBpdHMgcmVsaWFiaWxpdHk/CgpUaGlzIHdvbid0IG1ha2Ugc2Vuc2UgZm9yIGFsbCB0YXNrcy4gRm9yIGV4YW1wbGUgdG8gZXN0aW1hdGUgYSByaXNrIGF2ZXJzaW9uIHBhcmFtZXRlciB5b3UgbmVlZCBhbGwgdHJpYWxzIGZvciBIb2x0IGFuZCBMYXVyeS4gRm9yIEtpcmJ5IGFuZCBCaWNrZWwgeW91IGhhdmUgc3BlY2lmaWMgY29uZGl0aW9ucyBsb29raW5nIGF0IGZld2VyIHRyaWFscy4gVGhlIENvZ25pdGl2ZSBSZWZsZWN0aW9uIFRhc2sgbWlnaHQgYmUgbW9yZSBhcHByb3ByaWF0ZSB0byBhbmFseXplIGVhY2ggaXRlbSBzZWFwcmF0ZWx5LiBUaGUgd3JpdGluZyB0YXNrIGRvZXMgbm90IGhhdmUgdHJpYWwgbnVtYmVycy4gRm9yIGFsbCBvdGhlcnMgaXQgbWlnaHQgYmUgaW50ZXJlc3RpbmcgdG8gaW52ZXN0aWdhdGUuCgpUaGVzZSBraW5kcyBvZiBhbmFseXNlcyBhcmUgdG9vIHRhc2stc3BlY2lmaWMgYW5kIGluLWRlcHRoIGZvciBhIHBhcGVyIHRoYXQgaXMgdHJ5aW5nIHRvIGdpdmUgYSBnbG9iYWwgc2Vuc2Ugb2YgdGhlIGRpZmZlcmVuY2VzIGJldHdlZW4gc2VsZi1yZWd1bGF0aW9uIG1lYXN1cmVzIGluIHRoZWlyIHN1aXRhYmxpdHkgZm9yIGluZGl2aWR1YWwgZGlmZmVyZW5jZSBhbmFseXNlcyBiYXNlZCBvbiB0aGVpciBzdGFiaWxpdHkgYWNyb3NzIHRpbWUuIFN1Y2ggYW5hbHlzZXMgd291bGQgcHJvdmlkZSBhIGRldGFpbGVkIGV4YW1pbmF0aW9uIG9mIGhvdyB0byBleHRyYWN0IHRoZSBtb3N0IHJlbGlhYmxlL2Jlc3QgaW5kaXZpZHVhbCBkaWZmZXJlbmNlIG1lYXN1cmUgZnJvbSB0YXNrcyB3aXRoIGEgc2V0IG9mIG1lZGlvY3JlIHZhcmlhYmxlcyB0byBiZWdpbiB3aXRoLiBUaG91Z2ggd2UgZG8gbm90IHByb3ZpZGUgc3VjaCBhIGNvbXByZWhlbnNpdmUgYW5hbHlzaXMgb2YgdGhpcyBzb3J0IGluIHRoaXMgcGFwZXIgd2UgcHJvdmlkZSBhIHNpbmdsZSBleGFtcGxlIG9mIHRoaXMgYXBwcm9hY2ggYW5kIGhvcGUgdGhlIG9wZW4gYWNjZXNzIHdlIHByb3ZpZGUgdG8gdGhlIGRhdGEgc3B1cnMgZnVydGhlciB3b3JrLgoKRm9yIHRoaXMgZXhhbXBsZSB3ZSBsb29rIGF0IHRoZSByZXRlc3QgcmVsaWFiaWxpdHkgb2YgdGhlIHRocmVlIGRlcGVuZGVudCBtZWFzdXJlcyAoYXZlcmFnZSBhY2N1cmFjeSwgbWVkaWFuIHJlc3BvbnNlIHRpbWUsIHRvdGFsIHNjb3JlKSBmcm9tIHRoZSBoaWVyYXJjaGljYWwgcnVsZSB0YXNrIHdpdGggMzYwIHRyaWFscy4gSGVyZSBpcyBhIGdyYXBoIG9mIGhvdyB0aGUgcG9pbnQgZXN0aW1hdGVzIG9mIHRoZSByZXRlc3QgcmVsaWFiaWxpdHkgY2hhbmdlcyBmb3IgZWFjaCBvZiB0aGUgZGVwZW5kZW50IG1lYXN1cmVzIHVzaW5nIGRpZmZlcmVudCBudW1iZXJzIG9mIHRyaWFscyB0byBlc3RpbWF0ZSB0aGVtLiBXaGlsZSB0aGUgcmVsaWFiaWxpdHkgZXN0aW1hdGUgZm9yIGVhY2ggb2YgdGhlIHZhcmlhYmxlcyByZXNwZWN0aXZlbHkgYXJlIC4uLiwgLi4uIGFuZCAuLi4gdXNpbmcgYWxsIG9mIHRoZSB0cmlhbHMsIHRoZXNlIHZhbHVlcyBhcmUgaW5kZXBlbmRlbnQgb2YgbnVtYmVyIG9mIHRyaWFscyB1c2VkIGluIHRoZSB0YXNrIGZvciBvbmx5IGNlcnRhaW4gdmFyaWFibGVzLiBCYXNlZCBvbiB0aGlzIGFuYWx5c2lzIHJlc2VhcmNoZXJzIG1pZ2h0IGRlY2lkZSB0byB1c2UgYSB2ZXJzaW9uIG9mIHRoZSB0YXNrIHdpdGggZmV3ZXIgdHJpYWxzIG9yIG9ubHkgbWVhc3VyZXMgd2l0aCBjb25zaXN0ZW50bHkgaGlnaCByZWxpYWJpbGl0eSBlc3RpbWF0ZXMuCgpFeGFtcGxlIHRhc2s6IHRocmVlIGJ5IHR3bwoKUG9zdCBwcm9jZXNzIGR2J3MgZnJvbSBjbHVzdGVyIHRvIGNhbGN1bGF0ZSByZWxpYWJpbGl0aWVzCmBgYHtyfQp0MV9kdnMgPSByZWFkLmNzdihwYXN0ZTAocmV0ZXN0X2RhdGFfcGF0aCwgJ3QxX3RidF9kdnMuY3N2JykpCnQyX2R2cyA9IHJlYWQuY3N2KHBhc3RlMChyZXRlc3RfZGF0YV9wYXRoLCAndDJfdGJ0X2R2cy5jc3YnKSkKCnQxX2R2cyA9IHQxX2R2cyAlPiUgc2VsZWN0KC1YKQp0Ml9kdnMgPSB0Ml9kdnMgJT4lIHNlbGVjdCgtWCkKYGBgCgpgYGB7cn0KaHJfbWVyZ2UgPSBtZXJnZSh0MV9kdnMsIHQyX2R2cywgYnkgPSBjKCJzdWJfaWQiLCAiYnJlYWtzIikpCgpocl9tZXJnZSA9IGhyX21lcmdlICU+JQogIGdhdGhlcihrZXksIHZhbHVlLCAtc3ViX2lkLCAtYnJlYWtzKSAlPiUKICBzZXBhcmF0ZShrZXksIGMoImR2IiwgInRpbWUiKSwgc2VwPSJcXC4iKSAlPiUKICBtdXRhdGUodGltZSA9IGlmZWxzZSh0aW1lID09ICJ4IiwgMSwgMikpCgp0MV9kdnMgPSBocl9tZXJnZSAlPiUKICBmaWx0ZXIodGltZSA9PSAxKSAlPiUKICBzZWxlY3QoLXRpbWUpICU+JQogIHNwcmVhZChkdiwgdmFsdWUpCgp0Ml9kdnMgPSBocl9tZXJnZSAlPiUKICBmaWx0ZXIodGltZSA9PSAyKSAlPiUKICBzZWxlY3QoLXRpbWUpICU+JQogIHNwcmVhZChkdiwgdmFsdWUpCgojIGNhbGN1bGF0ZSBwb2ludCBlc3RpbWF0ZXMgZm9yIHJlbGlhYmlsaXR5IG9mIGVhY2ggb2YgdGhlIHZhcmlhYmxlcyBmb3IgZWFjaCBicmVhawojIGdldF9pY2MgZm9yIGVhY2ggYnJlYWsgb2YgdG1wX3QxX2R2cyBhbmQgdG1wX3QyX2R2cwoKdHJpYWxfbnVtX3JlbF9kZiA9IGRhdGEuZnJhbWUoYnJlYWtzPXJlcChOQSwgbGVuZ3RoKHVuaXF1ZSh0MV9kdnMkYnJlYWtzKSkpLAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICBhY2NfaWNjPXJlcChOQSwgbGVuZ3RoKHVuaXF1ZSh0MV9kdnMkYnJlYWtzKSkpLAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICBhdmdfcnRfZXJyb3JfaWNjPXJlcChOQSwgbGVuZ3RoKHVuaXF1ZSh0MV9kdnMkYnJlYWtzKSkpLAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICBzdGRfcnRfZXJyb3JfaWNjPXJlcChOQSwgbGVuZ3RoKHVuaXF1ZSh0MV9kdnMkYnJlYWtzKSkpLAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICBhdmdfcnRfaWNjPXJlcChOQSwgbGVuZ3RoKHVuaXF1ZSh0MV9kdnMkYnJlYWtzKSkpLAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICBzdGRfcnRfaWNjPXJlcChOQSwgbGVuZ3RoKHVuaXF1ZSh0MV9kdnMkYnJlYWtzKSkpLAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICBtaXNzZWRfcGVyY2VudF9pY2M9cmVwKE5BLCBsZW5ndGgodW5pcXVlKHQxX2R2cyRicmVha3MpKSksCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGN1ZV9zd2l0Y2hfY29zdF9ydF8xMDBfaWNjPXJlcChOQSwgbGVuZ3RoKHVuaXF1ZSh0MV9kdnMkYnJlYWtzKSkpLAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICBjdWVfc3dpdGNoX2Nvc3RfcnRfOTAwX2ljYz1yZXAoTkEsIGxlbmd0aCh1bmlxdWUodDFfZHZzJGJyZWFrcykpKSwKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgdGFza19zd2l0Y2hfY29zdF9ydF8xMDBfaWNjPXJlcChOQSwgbGVuZ3RoKHVuaXF1ZSh0MV9kdnMkYnJlYWtzKSkpLAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICB0YXNrX3N3aXRjaF9jb3N0X3J0XzkwMF9pY2M9cmVwKE5BLCBsZW5ndGgodW5pcXVlKHQxX2R2cyRicmVha3MpKSksCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGN1ZV9zd2l0Y2hfY29zdF9hY2NfMTAwX2ljYz1yZXAoTkEsIGxlbmd0aCh1bmlxdWUodDFfZHZzJGJyZWFrcykpKSwKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgY3VlX3N3aXRjaF9jb3N0X2FjY185MDBfaWNjPXJlcChOQSwgbGVuZ3RoKHVuaXF1ZSh0MV9kdnMkYnJlYWtzKSkpLAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICB0YXNrX3N3aXRjaF9jb3N0X2FjY18xMDBfaWNjPXJlcChOQSwgbGVuZ3RoKHVuaXF1ZSh0MV9kdnMkYnJlYWtzKSkpLAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICB0YXNrX3N3aXRjaF9jb3N0X2FjY185MDBfaWNjPXJlcChOQSwgbGVuZ3RoKHVuaXF1ZSh0MV9kdnMkYnJlYWtzKSkpKQoKZm9yKGkgaW4gMTpsZW5ndGgodW5pcXVlKHQxX2R2cyRicmVha3MpKSl7CiAgY3VyX2JyZWFrID0gdW5pcXVlKHQxX2R2cyRicmVha3MpW2ldCiAgdG1wX3QxX2R2cyA9IHQxX2R2cyAlPiUgZmlsdGVyKGJyZWFrcyA9PSBjdXJfYnJlYWspCiAgdG1wX3QyX2R2cyA9IHQyX2R2cyAlPiUgZmlsdGVyKGJyZWFrcyA9PSBjdXJfYnJlYWspCiAgdHJpYWxfbnVtX3JlbF9kZiRicmVha3NbaV0gPSBjdXJfYnJlYWsKICB0cmlhbF9udW1fcmVsX2RmJGFjY19pY2NbaV0gPSBnZXRfaWNjKCJhY2MiLCB0bXBfdDFfZHZzLCB0bXBfdDJfZHZzKQogIHRyaWFsX251bV9yZWxfZGYkYXZnX3J0X2Vycm9yX2ljY1tpXSA9IGdldF9pY2MoImF2Z19ydF9lcnJvciIsIHRtcF90MV9kdnMsIHRtcF90Ml9kdnMpCiAgdHJpYWxfbnVtX3JlbF9kZiRzdGRfcnRfZXJyb3JfaWNjW2ldID0gZ2V0X2ljYygic3RkX3J0X2Vycm9yIiwgdG1wX3QxX2R2cywgdG1wX3QyX2R2cykKICB0cmlhbF9udW1fcmVsX2RmJGF2Z19ydF9pY2NbaV0gPSBnZXRfaWNjKCJhdmdfcnQiLCB0bXBfdDFfZHZzLCB0bXBfdDJfZHZzKQogIHRyaWFsX251bV9yZWxfZGYkc3RkX3J0X2ljY1tpXSA9IGdldF9pY2MoInN0ZF9ydCIsIHRtcF90MV9kdnMsIHRtcF90Ml9kdnMpCiAgdHJpYWxfbnVtX3JlbF9kZiRtaXNzZWRfcGVyY2VudF9pY2NbaV0gPSBnZXRfaWNjKCJtaXNzZWRfcGVyY2VudCIsIHRtcF90MV9kdnMsIHRtcF90Ml9kdnMpCiAgdHJpYWxfbnVtX3JlbF9kZiRjdWVfc3dpdGNoX2Nvc3RfcnRfMTAwX2ljY1tpXSA9IGdldF9pY2MoImN1ZV9zd2l0Y2hfY29zdF9ydF8xMDAiLCB0bXBfdDFfZHZzLCB0bXBfdDJfZHZzKQogIHRyaWFsX251bV9yZWxfZGYkY3VlX3N3aXRjaF9jb3N0X3J0XzkwMF9pY2NbaV0gPSBnZXRfaWNjKCJjdWVfc3dpdGNoX2Nvc3RfcnRfOTAwIiwgdG1wX3QxX2R2cywgdG1wX3QyX2R2cykKICB0cmlhbF9udW1fcmVsX2RmJHRyaWFsX3N3aXRjaF9jb3N0X3J0XzEwMF9pY2NbaV0gPSBnZXRfaWNjKCJ0YXNrX3N3aXRjaF9jb3N0X3J0XzEwMCIsIHRtcF90MV9kdnMsIHRtcF90Ml9kdnMpCiAgdHJpYWxfbnVtX3JlbF9kZiR0cmlhbF9zd2l0Y2hfY29zdF9ydF85MDBfaWNjW2ldID0gZ2V0X2ljYygidGFza19zd2l0Y2hfY29zdF9ydF85MDAiLCB0bXBfdDFfZHZzLCB0bXBfdDJfZHZzKQogIHRyaWFsX251bV9yZWxfZGYkY3VlX3N3aXRjaF9jb3N0X2FjY18xMDBfaWNjW2ldID0gZ2V0X2ljYygiY3VlX3N3aXRjaF9jb3N0X2FjY18xMDAiLCB0bXBfdDFfZHZzLCB0bXBfdDJfZHZzKQogIHRyaWFsX251bV9yZWxfZGYkY3VlX3N3aXRjaF9jb3N0X2FjY185MDBfaWNjW2ldID0gZ2V0X2ljYygiY3VlX3N3aXRjaF9jb3N0X2FjY185MDAiLCB0bXBfdDFfZHZzLCB0bXBfdDJfZHZzKQogIHRyaWFsX251bV9yZWxfZGYkdHJpYWxfc3dpdGNoX2Nvc3RfYWNjXzEwMF9pY2NbaV0gPSBnZXRfaWNjKCJ0YXNrX3N3aXRjaF9jb3N0X2FjY18xMDAiLCB0bXBfdDFfZHZzLCB0bXBfdDJfZHZzKQogIHRyaWFsX251bV9yZWxfZGYkdHJpYWxfc3dpdGNoX2Nvc3RfYWNjXzkwMF9pY2NbaV0gPSBnZXRfaWNjKCJ0YXNrX3N3aXRjaF9jb3N0X2FjY185MDAiLCB0bXBfdDFfZHZzLCB0bXBfdDJfZHZzKQp9CnJtKGksIGN1cl9icmVhaywgdG1wX3QxX2R2cywgdG1wX3QyX2R2cykKCnRyaWFsX251bV9yZWxfZGYkYnJlYWtzID0gYXMubnVtZXJpYyh0cmlhbF9udW1fcmVsX2RmJGJyZWFrcykKCndyaXRlLmNzdih0cmlhbF9udW1fcmVsX2RmLCBwYXN0ZTAocmV0ZXN0X2RhdGFfcGF0aCwgJ3RyaWFsX251bV9yZWxfZGZfdGJ0LmNzdicpKQpgYGAKCmBgYHtyIHdhcm5pbmc9RkFMU0UsIG1lc3NhZ2U9RkFMU0V9CiMgdHJpYWxfbnVtX3JlbF9kZiA9IHJlYWQuY3N2KHBhc3RlMChyZXRlc3RfZGF0YV9wYXRoLCAndHJpYWxfbnVtX3JlbF9kZl90YnQuY3N2JykpCgpjb2xzIDwtIGMoIkFjY3VyYWN5IiA9ICcjMDg0NTk0JywgIkF2ZXJhZ2UgUlQgZXJyb3IiID0gJyM5OTAwMGQnLCAiU0QgUlQgZXJyb3IiID0gJyNjYjE4MWQnLCAiQXZlcmFnZSBSVCBjb3JyZWN0IiA9ICcjZWYzYjJjJywgIlNEIFJUIGNvcnJlY3QiID0gJyNmYjZhNGEnLCAiTWlzc2VkIHBlcmNlbnRhZ2UiID0gJyMyMTcxYjUnLCAiQ3VlIHN3aXRjaCBjb3N0IFJUICgxMDApIiA9ICcjZmM5MjcyJywgIkN1ZSBzd2l0Y2ggY29zdCBSVCAoOTAwKSIgPSAnI2ZjYmJhMScsICJUYXNrIHN3aXRjaCBjb3N0IFJUICgxMDApIiA9ICcjZmVlMGQyJywgIlRhc2sgc3dpdGNoIGNvc3QgUlQgKDkwMCkiID0gJyNmZmY1ZjAnLCAiQ3VlIHN3aXRjaCBjb3N0IEFjYyAoMTAwKSIgPSAnIzQyOTJjNicsICJDdWUgc3dpdGNoIGNvc3QgQWNjICg5MDApIiA9ICcjNmJhZWQ2JywgIlRhc2sgc3dpdGNoIGNvc3QgQWNjICgxMDApIiA9ICcjOWVjYWUxJywgIlRhc2sgc3dpdGNoIGNvc3QgQWNjICg5MDApIiA9ICcjYzZkYmVmJykKCnRyaWFsX251bV9yZWxfZGYgJT4lCiAgZ2F0aGVyKGtleSwgdmFsdWUsIC1icmVha3MpICU+JQogIGdncGxvdChhZXMoYnJlYWtzKjEwLCB2YWx1ZSwgY29sb3I9ZmFjdG9yKGtleSwgbGV2ZWxzID0gYygiYWNjX2ljYyIsICJhdmdfcnRfZXJyb3JfaWNjIiwgInN0ZF9ydF9lcnJvcl9pY2MiLCAiYXZnX3J0X2ljYyIsICJzdGRfcnRfaWNjIiwgIm1pc3NlZF9wZXJjZW50X2ljYyIsICJjdWVfc3dpdGNoX2Nvc3RfcnRfMTAwX2ljYyIsICJjdWVfc3dpdGNoX2Nvc3RfcnRfOTAwX2ljYyIsICJ0YXNrX3N3aXRjaF9jb3N0X3J0XzEwMF9pY2MiLCAidGFza19zd2l0Y2hfY29zdF9ydF85MDBfaWNjIiwgImN1ZV9zd2l0Y2hfY29zdF9hY2NfMTAwX2ljYyIsICJjdWVfc3dpdGNoX2Nvc3RfYWNjXzkwMF9pY2MiLCAidGFza19zd2l0Y2hfY29zdF9hY2NfMTAwX2ljYyIsICJ0YXNrX3N3aXRjaF9jb3N0X2FjY185MDBfaWNjIiksIGxhYmVscyA9IGMoIkFjY3VyYWN5IiwgIkF2ZXJhZ2UgUlQgZXJyb3IiLCAiU0QgUlQgZXJyb3IiLCAiQXZlcmFnZSBSVCBjb3JyZWN0IiwgIlNEIFJUIGNvcnJlY3QiLCAiTWlzc2VkIHBlcmNlbnRhZ2UiLCAiQ3VlIHN3aXRjaCBjb3N0IFJUICgxMDApIiwgIkN1ZSBzd2l0Y2ggY29zdCBSVCAoOTAwKSIsICJUYXNrIHN3aXRjaCBjb3N0IFJUICgxMDApIiwgIlRhc2sgc3dpdGNoIGNvc3QgUlQgKDkwMCkiLCAiQ3VlIHN3aXRjaCBjb3N0IEFjYyAoMTAwKSIsICJDdWUgc3dpdGNoIGNvc3QgQWNjICg5MDApIiwgIlRhc2sgc3dpdGNoIGNvc3QgQWNjICgxMDApIiwgIlRhc2sgc3dpdGNoIGNvc3QgQWNjICg5MDApIikpKSkrCiAgZ2VvbV9wb2ludCgpKwogIGdlb21fbGluZSgpKwogIHRoZW1lX2J3KCkrCiAgeGxhYigiTnVtYmVyIG9mIHRyaWFscyIpKwogIHlsYWIoIklDQyIpKwogIHRoZW1lKGxlZ2VuZC50aXRsZSA9IGVsZW1lbnRfYmxhbmsoKSwKICAgICAgICBsZWdlbmQucG9zaXRpb24gPSAnYm90dG9tJykrCiAgc2NhbGVfY29sb3JfbWFudWFsKHZhbHVlcyA9IGNvbHMsCiAgICAgICAgICAgICAgICAgICAgIGJyZWFrcyA9IGMoIkF2ZXJhZ2UgUlQgZXJyb3IiLCAiQWNjdXJhY3kiLCAiU0QgUlQgZXJyb3IiLCAiTWlzc2VkIHBlcmNlbnRhZ2UiLCAiQXZlcmFnZSBSVCBjb3JyZWN0IiwgIkN1ZSBzd2l0Y2ggY29zdCBBY2MgKDEwMCkiLCAiU0QgUlQgY29ycmVjdCIsICJDdWUgc3dpdGNoIGNvc3QgQWNjICg5MDApIiwgIkN1ZSBzd2l0Y2ggY29zdCBSVCAoMTAwKSIsICJUYXNrIHN3aXRjaCBjb3N0IEFjYyAoMTAwKSIsICJUYXNrIHN3aXRjaCBjb3N0IFJUICgxMDApIiwgIlRhc2sgc3dpdGNoIGNvc3QgQWNjICg5MDApIiwgIlRhc2sgc3dpdGNoIGNvc3QgUlQgKDkwMCkiKSkrCiAgeWxpbSgtMSwxKSsKICBndWlkZXMoY29sb3IgPSBndWlkZV9sZWdlbmQobmNvbD0yLCBieXJvdz1UKSkKCmdnc2F2ZSgnSW50cmFtZWFzdXJlX1RyaWFsbnVtX0RlcGVuZGVuZGVuY2UuanBnJywgZGV2aWNlID0gImpwZWciLCBwYXRoID0gIi9Vc2Vycy96ZXluZXBlbmthdmkvRHJvcGJveC9Qb2xkcmFja0xhYi9TUk9fUmV0ZXN0X0FuYWx5c2VzL291dHB1dC9maWd1cmVzIiwgd2lkdGggPSAxMCwgaGVpZ2h0ID0gNSwgdW5pdHMgPSAiaW4iLCBkcGkgPSAxMDApCmBgYAoKIyMjIFJhdyB2cyBERE0KCkNoZWNraW5nIERETSByZXN1bHRzIGluIHRoZSBib290c3RyYXBwZWQgZXN0aW1hdGVzLiBWYXJpYWJsZXMgdXNpbmcgYWxsIHRyaWFscyBhcmUgc2lnbmlmaWNhbnRseSBtb3JlIHJlbGlhYmxlIGNvbXBhcmVkIHRvIGRpZmZlcmVuY2Ugc2NvcmVzLiBSYXcgbWVhc3VyZXMgZG9uJ3QgZGlmZmVyIGZyb20gRERNIHBhcmFtZXRlcnMuIFdoaWNoIERETSBpcyBiZXR0ZXIgZGVwZW5kcyBvbiB3aGV0aGVyIGFsbCB0cmlhbHMgYXJlIHVzZWQuCgpgYGB7ciB3YXJuaW5nPUZBTFNFLCBtZXNzYWdlPUZBTFNFfQp0bXAgPSBtZWFzdXJlX2xhYmVscyAlPiUKICBtdXRhdGUoZHYgPSBhcy5jaGFyYWN0ZXIoZHYpKSAlPiUKICBmaWx0ZXIoZGRtX3Rhc2sgPT0gMSwgCiAgICAgICAgIG92ZXJhbGxfZGlmZmVyZW5jZSAhPSAiY29uZGl0aW9uIiwKICAgICAgICAgcnRfYWNjICE9ICdvdGhlcicpICU+JQogIGRyb3BfbmEoKSAlPiUKICBsZWZ0X2pvaW4oYm9vdF9kZlssYygiZHYiLCAiaWNjIiwgInNwZWFybWFuIildLCBieSA9ICdkdicpCgp0bXAgJT4lCiAgZHJvcF9uYSgpICU+JSAjdHJ5IHJlbW92aW5nIHRoaXMgaW4gZmluYWwgcmVsZWFzZQogIGdyb3VwX2J5KG92ZXJhbGxfZGlmZmVyZW5jZSwgcmF3X2ZpdCwgcnRfYWNjKSAlPiUKICBzdW1tYXJpc2UoaWNjX21lZGlhbiA9IHF1YW50aWxlKGljYywgcHJvYnMgPSAwLjUpLAogICAgICAgICAgICBpY2NfMi41ID0gcXVhbnRpbGUoaWNjLCBwcm9icyA9IDAuMDI1KSwKICAgICAgICAgICAgaWNjXzk3LjUgPSBxdWFudGlsZShpY2MsIHByb2JzID0gMC45NzUpLAogICAgICAgICAgICBzcGVhcm1hbl9tZWRpYW4gPSBxdWFudGlsZShzcGVhcm1hbiwgcHJvYnMgPSAwLjUpLAogICAgICAgICAgICBzcGVhcm1hbl8yLjUgPSBxdWFudGlsZShzcGVhcm1hbiwgcHJvYnMgPSAwLjAyNSksCiAgICAgICAgICAgIHNwZWFybWFuXzk3LjUgPSBxdWFudGlsZShzcGVhcm1hbiwgcHJvYnMgPSAwLjk3NSksCiAgICAgICAgICAgIG51bV92YXJzID0gbigpLzEwMDApICU+JQogIGRhdGF0YWJsZSgpICU+JQogIGZvcm1hdFJvdW5kKGNvbHVtbnM9YygnaWNjX21lZGlhbicsICdpY2NfMi41JywgJ2ljY185Ny41JywgJ3NwZWFybWFuX21lZGlhbicsICdzcGVhcm1hbl8yLjUnLCAnc3BlYXJtYW5fOTcuNScpLCBkaWdpdHM9MykKYGBgCgpgYGB7cn0KdG1wX3NhdmUgPSB0bXAgJT4lCiAgZHJvcF9uYSgpICU+JSAjdHJ5IHJlbW92aW5nIHRoaXMgaW4gZmluYWwgcmVsZWFzZQogIGdyb3VwX2J5KG92ZXJhbGxfZGlmZmVyZW5jZSwgcmF3X2ZpdCwgcnRfYWNjKSAlPiUKICBzdW1tYXJpc2UoaWNjX21lZGlhbiA9IHF1YW50aWxlKGljYywgcHJvYnMgPSAwLjUpLAogICAgICAgICAgICBpY2NfMi41ID0gcXVhbnRpbGUoaWNjLCBwcm9icyA9IDAuMDI1KSwKICAgICAgICAgICAgaWNjXzk3LjUgPSBxdWFudGlsZShpY2MsIHByb2JzID0gMC45NzUpLAogICAgICAgICAgICBudW1fdmFycyA9IG4oKS8xMDAwKSAlPiUKICB1bmdyb3VwKCkgJT4lCiAgbXV0YXRlKG92ZXJhbGxfZGlmZmVyZW5jZSA9IGFzLmNoYXJhY3RlcihvdmVyYWxsX2RpZmZlcmVuY2UpLAogICAgICAgICByYXdfZml0ID0gYXMuY2hhcmFjdGVyKHJhd19maXQpLAogICAgICAgICBydF9hY2MgPSBhcy5jaGFyYWN0ZXIocnRfYWNjKSkgJT4lCiAgYXJyYW5nZSgtaWNjX21lZGlhbikKCnNqdC5kZih0bXBfc2F2ZSAlPiUgbXV0YXRlX2lmKGlzLm51bWVyaWMsIGZ1bnMocm91bmQoLiwgMykpKSwgZGVzY3JpYmU9RiwgaGlkZS5wcm9ncmVzcyA9IFRSVUUsIHNob3cucm93bmFtZXMgPSBGQUxTRSwgZmlsZSA9ICIvVXNlcnMvemV5bmVwZW5rYXZpL0Ryb3Bib3gvUG9sZHJhY2tMYWIvU1JPX1JldGVzdF9BbmFseXNlcy9vdXRwdXQvdGFibGVzL2RkbV9yZWxfdGFibGUuZG9jIikKYGBgCgpDb21wYXJpbmcgb3ZlcmFsbCB2cyBkaWZmZXJlbmNlOiBvdmVyYWxsIGhhcyBoaWdoZXIgcmVsaWFiaWxpdHkgdGhhbiBkaWZmZXJlbmNlLgoKYGBge3J9CnN1bW1hcnkobG1lclRlc3Q6OmxtZXIoaWNjIH4gb3ZlcmFsbF9kaWZmZXJlbmNlICsgKDF8ZHYpICx0bXApKQpgYGAKCiJjb25zaXN0ZW50IHdpdGggdGhlIHN1bW1pbmcgb2YgdmFyaWFuY2UgaW4gdGhlIGRpZmZlcmVuY2Ugc2NvcmUiCgpgYGB7cn0KdG1wID0gbWVhc3VyZV9sYWJlbHMgJT4lCiAgbXV0YXRlKGR2ID0gYXMuY2hhcmFjdGVyKGR2KSkgJT4lCiAgZmlsdGVyKGRkbV90YXNrID09IDEsIAogICAgICAgICBvdmVyYWxsX2RpZmZlcmVuY2UgIT0gImNvbmRpdGlvbiIsCiAgICAgICAgIHJ0X2FjYyAhPSAnb3RoZXInKSAlPiUKICBkcm9wX25hKCkgJT4lCiAgbGVmdF9qb2luKGJvb3RfZGZbLGMoImR2IiwgImljYyIsICJ2YXJfc3VicyIsInZhcl9pbmQiLCAidmFyX3Jlc2lkIildLCBieSA9ICdkdicpICU+JQogIG11dGF0ZSh2YXJfc3Vic19wY3QgPSB2YXJfc3Vicy8odmFyX3N1YnMrdmFyX2luZCt2YXJfcmVzaWQpLAogICAgICAgICB2YXJfaW5kX3BjdCA9IHZhcl9pbmQvKHZhcl9zdWJzK3Zhcl9pbmQrdmFyX3Jlc2lkKSwgCiAgICAgICAgIHZhcl9yZXNpZF9wY3QgPSB2YXJfcmVzaWQvKHZhcl9zdWJzK3Zhcl9pbmQrdmFyX3Jlc2lkKSkgJT4lCiAgc2VsZWN0KC10YXNrLCAtZGRtX3Rhc2ssIC1udW1fYWxsX3RyaWFscywgLXZhcl9yZXNpZCwgLXZhcl9zdWJzLCAtdmFyX2luZCkgJT4lCiAgZ2F0aGVyKGtleSwgdmFsdWUsIC1kdiwgLW92ZXJhbGxfZGlmZmVyZW5jZSwgLXJhd19maXQsIC1ydF9hY2MsIC1pY2MpCgp0bXAgJT4lCiAgZmlsdGVyKGtleSA9PSAidmFyX3Jlc2lkX3BjdCIpICU+JQogIGdncGxvdChhZXMocmF3X2ZpdCwgdmFsdWUsIGZpbGw9b3ZlcmFsbF9kaWZmZXJlbmNlKSkrCiAgZ2VvbV9ib3hwbG90KCkKICAKc3VtbWFyeShsbWVyKHZhbHVlICB+IG92ZXJhbGxfZGlmZmVyZW5jZSpyYXdfZml0ICsgKDF8ZHYpLCBkYXRhPXRtcCAlPiUgZmlsdGVyKGtleT09InZhcl9yZXNpZF9wY3QiKSkpCgojU2ltcGxlIGVmZmVjdHMKc3VtbWFyeShsbWVyKHZhbHVlICB+IG92ZXJhbGxfZGlmZmVyZW5jZSArICgxfGR2KSwgZGF0YT10bXAgJT4lIGZpbHRlcihrZXk9PSJ2YXJfcmVzaWRfcGN0IiwgcmF3X2ZpdD09ICJFWiIpKSkKCnN1bW1hcnkobG1lcih2YWx1ZSAgfiBvdmVyYWxsX2RpZmZlcmVuY2UgKyAoMXxkdiksIGRhdGE9dG1wICU+JSBmaWx0ZXIoa2V5PT0idmFyX3Jlc2lkX3BjdCIsIHJhd19maXQ9PSAicmF3IikpKQoKc3VtbWFyeShsbWVyKHZhbHVlICB+IG92ZXJhbGxfZGlmZmVyZW5jZSArICgxfGR2KSwgZGF0YT10bXAgJT4lIGZpbHRlcihrZXk9PSJ2YXJfcmVzaWRfcGN0IiwgcmF3X2ZpdD09ICJoZGRtIikpKQpgYGAKCkNoZWNraW5nIFJvZ29zYSdzIGNsYWltIHRoYXQg4oCYRGlmZmVyZW5jZSBzY29yZXMgYXJlIHJlbGlhYmxlIHdoZW4gaW5kaXZpZHVhbCBkaWZmZXJlbmNlcyBpbiB0cnVlIGNoYW5nZSBleGlzdC7igJkKSG93IGRvIHdlIG1lYXN1cmUg4oCYaW5kaXZpZHVhbCBkaWZmZXJlbmNlIGluIHRydWUgY2hhbmdl4oCZIAoxLmJldHdlZW4gc3ViamVjdCB2YXJpYWJpbGl0eSBpbiBkaWZmZXJlbmNlIHNjb3JlIGRpc3RyaWJ1dGlvbnMKMi53aXRoaW4gc3ViamVjdCB2YXJpYW5jZSBpbiBpY2MKVXNpbmcgMiAoYmVjYXVzZSBjb21wYXJhYmxlIGJldHdlZW4gc3ViamVjdCB2YXJpYW5jZSBhY3Jvc3MgZGlmZmVyZW50IG1lYXN1cmVzIHdvdWxkIGJlIGhhcmQpCkV4cGVjdGluZyB0byBzZWU6IHBvc2l0aXZlIGNvcnJlbGF0aW9uIGJldHdlZW4gbWVhbiBpY2MgYW5kIG1lYW4gdmFyX2luZF9wY3QKUmVzdWx0OiBUaGUgY29ycmVsYXRpb24gYmV0d2VlbiBtZWFuIGljYyBhbmQgbWVhbiB2YXIgaW5kIHBjdCBpcyBOT1Qgc2lnbmlmaWNhbnQgQlVUIGxvb2tpbmcgYXQgdGhlIGRpc3RyaWJ1dGlvbiBvZiBtZWFuIHZhciBpbmQgcGN0IHRoZXJlIGlzIG5vdCBhIGxvdCBvZiB2YXJpYWJpbGl0eSB0CgpgYGB7cn0KdG1wID0gdG1wICU+JQogIHNwcmVhZChrZXksIHZhbHVlKSAlPiUKICBncm91cF9ieShkdikgJT4lCiAgc3VtbWFyaXNlKG1lYW5fdmFyX2luZF9wY3QgPSBtZWFuKHZhcl9pbmRfcGN0KSwKICAgICAgICAgICAgbWVhbl92YXJfc3Vic19wY3QgPSBtZWFuKHZhcl9zdWJzX3BjdCksCiAgICAgICAgICAgIG1lYW5faWNjID0gbWVhbihpY2MpKSAlPiUKICBsZWZ0X2pvaW4obWVhc3VyZV9sYWJlbHMsIGJ5PSdkdicpCgp0bXAgJT4lCiAgZmlsdGVyKG92ZXJhbGxfZGlmZmVyZW5jZSA9PSAnZGlmZmVyZW5jZScpICU+JQogIGdncGxvdChhZXMobWVhbl92YXJfaW5kX3BjdCwgbWVhbl9pY2MpKSsKICBnZW9tX3BvaW50KCkrCiAgZ2VvbV9hYmxpbmUoc2xvcGU9MSwgaW50ZXJjZXB0PTApKwogIGdlb21fc21vb3RoKG1ldGhvZCA9ICJsbSIpCgpzdW1tYXJ5KGxtKG1lYW5faWNjIH4gbWVhbl92YXJfaW5kX3BjdCxkYXRhPXRtcCAlPiUKICBmaWx0ZXIob3ZlcmFsbF9kaWZmZXJlbmNlID09ICdkaWZmZXJlbmNlJykpKQoKdG1wICU+JQogIGZpbHRlcihvdmVyYWxsX2RpZmZlcmVuY2UgPT0gJ2RpZmZlcmVuY2UnKSAlPiUKICBnZ3Bsb3QoYWVzKG1lYW5fdmFyX2luZF9wY3QpKSsKICBnZW9tX2RlbnNpdHkoKQoKc3VtbWFyeSh0bXAkbWVhbl92YXJfaW5kX3BjdFt0bXAkb3ZlcmFsbF9kaWZmZXJlbmNlID09ICJkaWZmZXJlbmNlIl0pCnN1bW1hcnkodG1wJG1lYW5fdmFyX3N1YnNfcGN0W3RtcCRvdmVyYWxsX2RpZmZlcmVuY2UgPT0gImRpZmZlcmVuY2UiXSkKCnRtcCA9IHRtcCAlPiUgZmlsdGVyKG92ZXJhbGxfZGlmZmVyZW5jZSA9PSAnZGlmZmVyZW5jZScpCnQudGVzdCh0bXAkbWVhbl92YXJfaW5kX3BjdCwgdG1wJG1lYW5fdmFyX3N1YnNfcGN0LCBwYWlyZWQ9VCkKYGBgCgpgYGB7cn0KdG1wID0gbWVhc3VyZV9sYWJlbHMgJT4lCiAgbXV0YXRlKGR2ID0gYXMuY2hhcmFjdGVyKGR2KSkgJT4lCiAgZmlsdGVyKGRkbV90YXNrID09IDEsIAogICAgICAgICBvdmVyYWxsX2RpZmZlcmVuY2UgIT0gImNvbmRpdGlvbiIsCiAgICAgICAgIHJ0X2FjYyAhPSAnb3RoZXInKSAlPiUKICBkcm9wX25hKCkgJT4lCiAgbGVmdF9qb2luKGJvb3RfZGZbLGMoImR2IiwgImljYyIsICJzcGVhcm1hbiIpXSwgYnkgPSAnZHYnKQpgYGAKCkNvbXBhcmluZyByYXcgdnMgZGRtIGluIG92ZXJhbGwgZXN0aW1hdGVzOiBFWiBpcyBzaWduaWZpY2FudGx5IGJldHRlciB0aGFuIEhERE0gYW5kIGNvbXBhcmFibGUgdG8gcmF3IGVzdGltYXRlcy4KCmBgYHtyfQpzdW1tYXJ5KGxtZXJUZXN0OjpsbWVyKGljYyB+IHJhd19maXQgKyAoMXxkdikgLHRtcCAlPiUgZmlsdGVyKG92ZXJhbGxfZGlmZmVyZW5jZSA9PSAib3ZlcmFsbCIpKSkKYGBgCgpDb21wYXJpbmcgcmF3IHZzIGRkbSBpbiBkaWZmZXJlbmNlIHNjb3JlczogRVogaXMgc2lnbmlmaWNhbnRseSB3b3JzZSB0aGFuIEhERE0gYW5kIGNvbXBhcmFibGUgdG8gcmF3IGVzdGltYXRlcy4KCmBgYHtyfQpzdW1tYXJ5KGxtZXJUZXN0OjpsbWVyKGljYyB+IHJhd19maXQgKyAoMXxkdikgLHRtcCAlPiUgZmlsdGVyKG92ZXJhbGxfZGlmZmVyZW5jZSA9PSAiZGlmZmVyZW5jZSIpKSkKYGBgCgpgYGB7cn0KdG1wICU+JQogIGdncGxvdChhZXMoZmFjdG9yKHJhd19maXQsIGxldmVscyA9IGMoInJhdyIsICJFWiIsICJoZGRtIiksIGxhYmVscz1jKCJSYXciLCAiRVotZGlmZnVzaW9uIiwgIkhpZXJhcmNoaWNhbCBkaWZmdXNpb24iKSksIGljYywgZmlsbD1mYWN0b3IocnRfYWNjLCBsZXZlbHMgPSBjKCJydCIsImFjY3VyYWN5IiwgImRyaWZ0IHJhdGUiLCAidGhyZXNob2xkIiwgIm5vbi1kZWNpc2lvbiIpLCBsYWJlbHM9YygiUmVzcG9uc2UgVGltZSIsICJBY2N1cmFjeSIsIkRyaWZ0IFJhdGUiLCAiVGhyZXNob2xkIiwgIk5vbi1kZWNpc2lvbiIpKSkpKwogIGdlb21fYm94cGxvdCgpKwogIGZhY2V0X3dyYXAofmZhY3RvcihvdmVyYWxsX2RpZmZlcmVuY2UsIGxldmVscz1jKCJvdmVyYWxsIiwgImRpZmZlcmVuY2UiKSwgbGFiZWxzPWMoIk92ZXJhbGwiLCAiRGlmZmVyZW5jZSIpKSkrCiAgdGhlbWVfYncoKSsKICB5bGFiKCJJQ0MiKSsKICB4bGFiKCIiKSsKICB0aGVtZShsZWdlbmQudGl0bGUgPSBlbGVtZW50X2JsYW5rKCksCiAgICAgICAgbGVnZW5kLnBvc2l0aW9uID0gJ2JvdHRvbScsCiAgICAgICAgbGVnZW5kLnRleHQgPSBlbGVtZW50X3RleHQoc2l6ZT0xNiksCiAgICAgICAgc3RyaXAudGV4dCA9IGVsZW1lbnRfdGV4dChzaXplPTE2KSwKICAgICAgICBheGlzLnRleHQgPSBlbGVtZW50X3RleHQoc2l6ZSA9IDE2KSwKICAgICAgICB0ZXh0ID0gZWxlbWVudF90ZXh0KHNpemU9MTYpKSsKICBndWlkZXMoZmlsbCA9IGd1aWRlX2xlZ2VuZChuY29sID0gMikpKwogIHNjYWxlX2ZpbGxfZGlzY3JldGUoYnJlYWtzPWMoIkRyaWZ0IFJhdGUiLCAiVGhyZXNob2xkIiwgIk5vbi1kZWNpc2lvbiIsICJSZXNwb25zZSBUaW1lIiwgIkFjY3VyYWN5IikpCgpnZ3NhdmUoJ0Jvb3RzdHJhcF9ERE1fQ29tcC5qcGcnLCBkZXZpY2UgPSAianBlZyIsIHBhdGggPSAiL1VzZXJzL3pleW5lcGVua2F2aS9Ecm9wYm94L1BvbGRyYWNrTGFiL1NST19SZXRlc3RfQW5hbHlzZXMvb3V0cHV0L2ZpZ3VyZXMvIiwgd2lkdGggPSAxNCwgaGVpZ2h0ID0gMTAsIHVuaXRzID0gImluIiwgbGltaXRzaXplID0gRkFMU0UpCmBgYAoKYGBge3J9CnRtcDIgPSBtZWFzdXJlX2xhYmVscyAlPiUKICBtdXRhdGUoZHYgPSBhcy5jaGFyYWN0ZXIoZHYpKSAlPiUKICBmaWx0ZXIoZGRtX3Rhc2sgPT0gMSwgCiAgICAgICAgIG92ZXJhbGxfZGlmZmVyZW5jZSAhPSAiY29uZGl0aW9uIiwKICAgICAgICAgcnRfYWNjICE9ICJvdGhlciIpICU+JQogIGRyb3BfbmEoKSAlPiUKICBsZWZ0X2pvaW4ocmVsX2RmWyxjKCJkdiIsICJpY2MiLCAic3BlYXJtYW4iKV0sIGJ5ID0gJ2R2JykKCnRtcDIgJT4lCiAgZ2dwbG90KGFlcyhmYWN0b3IocmF3X2ZpdCwgbGV2ZWxzID0gYygicmF3IiwgIkVaIiwgImhkZG0iKSwgbGFiZWxzPWMoIlJhdyIsICJFWi1kaWZmdXNpb24iLCAiSGllcmFyY2hpY2FsIGRpZmZ1c2lvbiIpKSwgaWNjLCBmaWxsPWZhY3RvcihydF9hY2MsIGxldmVscyA9IGMoInJ0IiwiYWNjdXJhY3kiLCAiZHJpZnQgcmF0ZSIsICJ0aHJlc2hvbGQiLCAibm9uLWRlY2lzaW9uIiksIGxhYmVscz1jKCJSZXNwb25zZSBUaW1lIiwgIkFjY3VyYWN5IiwiRHJpZnQgUmF0ZSIsICJUaHJlc2hvbGQiLCAiTm9uLWRlY2lzaW9uIikpKSkrCiAgZ2VvbV9ib3hwbG90KCkrCiAgZmFjZXRfd3JhcCh+ZmFjdG9yKG92ZXJhbGxfZGlmZmVyZW5jZSwgbGV2ZWxzPWMoIm92ZXJhbGwiLCAiZGlmZmVyZW5jZSIpLCBsYWJlbHM9YygiT3ZlcmFsbCIsICJEaWZmZXJlbmNlIikpKSsKICB0aGVtZV9idygpKwogIHlsYWIoIklDQyIpKwogIHhsYWIoIiIpKwogIHRoZW1lKGxlZ2VuZC50aXRsZSA9IGVsZW1lbnRfYmxhbmsoKSwKICAgICAgICBsZWdlbmQucG9zaXRpb24gPSAnYm90dG9tJywKICAgICAgICBsZWdlbmQudGV4dCA9IGVsZW1lbnRfdGV4dChzaXplPTE2KSwKICAgICAgICBzdHJpcC50ZXh0ID0gZWxlbWVudF90ZXh0KHNpemU9MTYpLAogICAgICAgIGF4aXMudGV4dCA9IGVsZW1lbnRfdGV4dChzaXplID0gMTYpLAogICAgICAgIHRleHQgPSBlbGVtZW50X3RleHQoc2l6ZT0xNikpKwogIGd1aWRlcyhmaWxsID0gZ3VpZGVfbGVnZW5kKG5jb2wgPSAyKSkrCiAgc2NhbGVfZmlsbF9kaXNjcmV0ZShicmVha3M9YygiRHJpZnQgUmF0ZSIsICJUaHJlc2hvbGQiLCAiTm9uLWRlY2lzaW9uIiwgIlJlc3BvbnNlIFRpbWUiLCAiQWNjdXJhY3kiKSkKCmdnc2F2ZSgnUG9pbnRFc3RfRERNX0NvbXAuanBnJywgZGV2aWNlID0gImpwZWciLCBwYXRoID0gIi9Vc2Vycy96ZXluZXBlbmthdmkvRHJvcGJveC9Qb2xkcmFja0xhYi9TUk9fUmV0ZXN0X0FuYWx5c2VzL291dHB1dC9maWd1cmVzLyIsIHdpZHRoID0gMTQsIGhlaWdodCA9IDEwLCB1bml0cyA9ICJpbiIsIGxpbWl0c2l6ZSA9IEZBTFNFKQpgYGAKCiMjIFN1cnZleSByZWxpYWJpbGl0aWVzCgpgYGB7ciB3YXJuaW5nPUZBTFNFLCBtZXNzYWdlPUZBTFNFfQp0bXAgPSBtZWFzdXJlX2xhYmVscyAlPiUKICBtdXRhdGUoZHYgPSBhcy5jaGFyYWN0ZXIoZHYpKSAlPiUKICBmaWx0ZXIodGFzayA9PSAnc3VydmV5JykgJT4lCiAgbGVmdF9qb2luKGJvb3RfZGZbLGMoImR2IiwgImljYyIsICJzcGVhcm1hbiIpXSwgYnkgPSAnZHYnKSAlPiUKICBmaWx0ZXIob3ZlcmFsbF9kaWZmZXJlbmNlICE9ICdkaWZmZXJlbmNlJyAmIHJhd19maXQgJWluJSBjKCdFWicsICdoZGRtJykgPT0gRkFMU0UpJT4lCiAgc2VwYXJhdGUoZHYsIGMoJ3Rhc2tfbmFtZScsICdleHRyYV8xJywgJ2V4dHJhXzInKSwgc2VwID0gJ1xcLicscmVtb3ZlPUZBTFNFKSAlPiUKICBzZWxlY3QoLWV4dHJhXzEsIC1leHRyYV8yKSAlPiUKICBncm91cF9ieSh0YXNrX25hbWUpICU+JQogIHN1bW1hcmlzZShtZWRpYW5faWNjID0gbWVkaWFuKGljYyksCiAgICAgICAgICAgIG1lZGlhbl9zcGVhcm1hbiA9IG1lZGlhbihzcGVhcm1hbiksCiAgICAgICAgICAgIGljY18yLjUgPSBxdWFudGlsZShpY2MsIHByb2JzID0gMC4wMjUpLAogICAgICAgICAgICBpY2NfOTcuNSA9IHF1YW50aWxlKGljYywgcHJvYnMgPSAwLjk3NSksCiAgICAgICAgICAgIG1pbl9zcGVhcm1hbiA9IG1pbihzcGVhcm1hbiksCiAgICAgICAgICAgIG1heF9zcGVhcm1hbiA9IG1heChzcGVhcm1hbiksCiAgICAgICAgICAgIG51bV9tZWFzdXJlcyA9IG4oKS8xMDAwLAogICAgICAgICAgICBtZWFuX251bV90cmlhbHMgPSByb3VuZChtZWFuKG51bV9hbGxfdHJpYWxzKSkpJT4lCiAgYXJyYW5nZSgtbWVkaWFuX2ljYywgLW1lZGlhbl9zcGVhcm1hbikKCnRtcCAlPiUKICBkYXRhdGFibGUoKSAlPiUKICBmb3JtYXRSb3VuZChjb2x1bW5zPWMoJ21lZGlhbl9zcGVhcm1hbicsICdtZWRpYW5faWNjJywKICAgICAgICAgICAgICAgICAgICAgICAgJ21pbl9zcGVhcm1hbicsICdpY2NfMi41JywKICAgICAgICAgICAgICAgICAgICAgICAgJ21heF9zcGVhcm1hbicsICdpY2NfOTcuNScpLCBkaWdpdHM9MykKYGBgCgpgYGB7cn0KdG1wID0gdG1wJT4lCiAgc2VsZWN0KC1taW5fc3BlYXJtYW4sIC1tYXhfc3BlYXJtYW4sLW1lZGlhbl9zcGVhcm1hbikgJT4lCiAgbXV0YXRlKHRhc2tfbmFtZSA9IGdzdWIoIl8iLCAiICIsIHRhc2tfbmFtZSksCiAgICAgICAgIHRhc2tfbmFtZSA9IGdzdWIoIihefFtbOnNwYWNlOl1dKShbWzphbHBoYTpdXSkiLCAiXFwxXFxVXFwyIiwgdGFza19uYW1lLCBwZXJsPVRSVUUpLAogICAgICAgICB0YXNrX25hbWUgPSBnc3ViKCJTdXJ2ZXkiLCAiIiwgdGFza19uYW1lKSkKCm5hbWVzKHRtcCkgPSBnc3ViKCJfIiwgIiAiLCBuYW1lcyh0bXApKQpuYW1lcyh0bXApID0gZ3N1YigiKF58W1s6c3BhY2U6XV0pKFtbOmFscGhhOl1dKSIsICJcXDFcXFVcXDIiLCBuYW1lcyh0bXApLCBwZXJsPVRSVUUpCgpzanQuZGYodG1wICU+JSBtdXRhdGVfaWYoaXMubnVtZXJpYywgZnVucyhyb3VuZCguLCAzKSkpLCBkZXNjcmliZT1GLCBoaWRlLnByb2dyZXNzID0gVFJVRSwgc2hvdy5yb3duYW1lcyA9IEZBTFNFLCBmaWxlID0gIi9Vc2Vycy96ZXluZXBlbmthdmkvRHJvcGJveC9Qb2xkcmFja0xhYi9TUk9fUmV0ZXN0X0FuYWx5c2VzL291dHB1dC90YWJsZXMvc3VydmV5X3JlbF90YWJsZS5kb2MiKQpgYGAKCiMjIyBOdW1iZXIgb2YgaXRlbXMKCkRvZXMgbnVtYmVyIG9mIGl0ZW1zIGluIGEgc3VydmV5IGhhdmUgYSBzaWduaWZpY2FudCBlZmZlY3Qgb24gdGhlIGF2ZXJhZ2UgSUNDIG9mIHN1cnZleSBtZWFzdXJlcz8gTm8uIAoKYGBge3Igd2FybmluZz1GQUxTRSwgbWVzc2FnZT1GQUxTRX0KdG1wID0gbWVhc3VyZV9sYWJlbHMgJT4lCiAgbXV0YXRlKGR2ID0gYXMuY2hhcmFjdGVyKGR2KSkgJT4lCiAgZmlsdGVyKHRhc2sgPT0gJ3N1cnZleScpICU+JQogICMgbGVmdF9qb2luKHJlbF9kZlssYygiZHYiLCAic3BlYXJtYW4iLCJpY2MiKV0sIGJ5PSdkdicpICU+JQogIGxlZnRfam9pbihib290X2RmWyxjKCJkdiIsICJzcGVhcm1hbiIsImljYyIpXSwgYnk9J2R2JykgJT4lCiAgZmlsdGVyKG92ZXJhbGxfZGlmZmVyZW5jZSAhPSAnZGlmZmVyZW5jZScgJiByYXdfZml0ICVpbiUgYygnRVonLCAnaGRkbScpID09IEZBTFNFKSU+JQogIHNlcGFyYXRlKGR2LCBjKCd0YXNrX25hbWUnLCAnZXh0cmFfMScsICdleHRyYV8yJyksIHNlcCA9ICdcXC4nLHJlbW92ZT1GQUxTRSkgJT4lCiAgc2VsZWN0KC1leHRyYV8xLCAtZXh0cmFfMikKCiMgc3VtbWFyeShsbShpY2MgfiBudW1fYWxsX3RyaWFscywgZGF0YSA9IHRtcCkpCnN1bW1hcnkobG1lclRlc3Q6OmxtZXIoaWNjIH4gbnVtX2FsbF90cmlhbHMgKyAoMXxkdiksIGRhdGEgPSB0bXApKQpgYGAKCmBgYHtyIHdhcm5pbmc9RkFMU0UsIG1lc3NhZ2U9RkFMU0V9Cm1lYXN1cmVfbGFiZWxzICU+JQogIG11dGF0ZShkdiA9IGFzLmNoYXJhY3RlcihkdikpICU+JQogIGZpbHRlcih0YXNrID09ICd0YXNrJykgJT4lCiAgIyBsZWZ0X2pvaW4ocmVsX2RmWyxjKCJkdiIsICJzcGVhcm1hbiIsImljYyIpXSwgYnk9J2R2JykgJT4lCiAgbGVmdF9qb2luKGJvb3RfZGZbLGMoImR2IiwgInNwZWFybWFuIiwiaWNjIildLCBieT0nZHYnKSAlPiUKICBmaWx0ZXIob3ZlcmFsbF9kaWZmZXJlbmNlICE9ICdkaWZmZXJlbmNlJyAmIHJhd19maXQgJWluJSBjKCdFWicsICdoZGRtJykgPT0gRkFMU0UpJT4lCiAgc2VwYXJhdGUoZHYsIGMoJ3Rhc2tfbmFtZScsICdleHRyYV8xJywgJ2V4dHJhXzInKSwgc2VwID0gJ1xcLicscmVtb3ZlPUZBTFNFKSAlPiUKICBzZWxlY3QoLWV4dHJhXzEsIC1leHRyYV8yKSAlPiUKICBnZ3Bsb3QoYWVzKG51bV9hbGxfdHJpYWxzLCBpY2MpKSsKICBnZW9tX3BvaW50KCkrCiAgZ2VvbV9zbW9vdGgobWV0aG9kPSJsbSIpKwogIHRoZW1lX2J3KCkrCiAgeGxhYigiTnVtYmVyIG9mIHRyaWFscyIpKwogIHlsYWIoIklDQyIpCmBgYAo=